Skip to content

Commit cfdd645

Browse files
IEvangelisttdykstra
IEvangelist
and
tdykstra
authored
New article that details unit test code coverage using coverlet (#18955)
* Initial bits of new article * Fix linting issue * Correct link * Updates * Final updates * Fix md error * Apply suggestions from code review Co-authored-by: Tom Dykstra <tdykstra@microsoft.com> * Many more updates * Massive updates, more like a tutorial now * Added a lot and managed updates * Minor tweaks * Updates for acrolinx * Another minor update * Apply suggestions from code review Co-authored-by: Tom Dykstra <tdykstra@microsoft.com> * Updates from peer review. * Proper casing * Final change Co-authored-by: Tom Dykstra <tdykstra@microsoft.com>
1 parent 876af04 commit cfdd645

File tree

5 files changed

+310
-1
lines changed

5 files changed

+310
-1
lines changed

‎docs/core/testing/media/test-report.png‎

92.6 KB
Loading

‎docs/core/testing/order-unit-tests.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,4 +79,4 @@ To order tests explicitly, NUnit provides an [`OrderAttribute`](https://github.c
7979
## Next Steps
8080

8181
> [!div class="nextstepaction"]
82-
> [Unit testing best practices](unit-testing-best-practices.md)
82+
> [Unit test code coverage](unit-testing-code-coverage.md)

‎docs/core/testing/unit-testing-best-practices.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,12 @@ Writing tests for your code will naturally decouple your code, because it would
4444
- **Self-Checking**. The test should be able to automatically detect if it passed or failed without any human interaction.
4545
- **Timely**. A unit test should not take a disproportionately long time to write compared to the code being tested. If you find testing the code taking a large amount of time compared to writing the code, consider a design that is more testable.
4646

47+
## Code coverage
48+
49+
A high code coverage percentage is often associated with a higher quality of code. However, the measurement itself *cannot* determine the quality of code. Setting an overly ambitious code coverage percentage goal can be counterproductive. Imagine a complex project with thousands of conditional branches, and imagine that you set a goal of 95% code coverage. Currently the project maintains 90% code coverage. The amount of time it takes to account for all of the edge cases in the remaining 5% could be a massive undertaking, and the value proposition quickly diminishes.
50+
51+
A high code coverage percentage is not an indicator of success, nor does it imply high code quality. It jusst represents the amount of code that is covered by unit tests. For more information, see [unit testing code coverage](unit-testing-code-coverage.md).
52+
4753
## Let's speak the same language
4854
The term *mock* is unfortunately very misused when talking about testing. The following defines the most common types of *fakes* when writing unit tests:
4955

‎docs/core/testing/unit-testing-code-coverage.md‎

Lines changed: 301 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,301 @@
1+
---
2+
title: Use code coverage for unit testing
3+
description: Learn how to use the code coverage capabilities for .NET unit tests.
4+
author: IEvangelist
5+
ms.author: dapine
6+
ms.date: 06/16/2020
7+
---
8+
9+
# Use code coverage for unit testing
10+
11+
Unit tests help to ensure functionality, and provide a means of verification for refactoring efforts. Code coverage is a measurement of the amount of code that is run by unit tests - either lines, branches, or methods. As an example, if you have a simple application with only two conditional branches of code (_branch a_, and _branch b_), a unit test that verifies conditional _branch a_ will report branch code coverage of 50%.
12+
13+
This article discusses the usage of code coverage for unit testing with Coverlet and report generation using ReportGenerator. While this article focuses on C# and xUnit as the test framework, both MSTest and NUnit would also work. Coverlet is an [open source project on GitHub](https://github.com/coverlet-coverage/coverlet) that provides a cross platform code coverage framework for C#. [Coverlet](https://dotnetfoundation.org/projects/coverlet) is part of the .NET foundation. Coverlet collects Cobertura coverage test run data, which is used for report generation.
14+
15+
Additionally, this article details how to use the code coverage information collected from a Coverlet test run to generate a report. The report generation is possible using another [open source project on GitHub - ReportGenerator](https://github.com/danielpalme/ReportGenerator). ReportGenerator converts coverage reports generated by Cobertura among many others, into human readable reports in various formats.
16+
17+
## System under test
18+
19+
The "system under test" refers to the code that you're writing unit tests against, this could be an object, service, or anything else that exposes testable functionality. For the purpose of this article, you'll create a class library that will be the system under test, and two corresponding unit test projects.
20+
21+
### Create a class library
22+
23+
From a command prompt in a new directory named `UnitTestingCodeCoverage`, create a new .NET standard class library using the [`dotnet new classlib`](../tools/dotnet-new.md#classlib) command:
24+
25+
```dotnetcli
26+
dotnet new classlib -n Numbers
27+
```
28+
29+
The snippet below defines a simple `PrimeService` class that provides functionality to check if a number is prime. Copy the snippet below and replace the contents of the *Class1.cs* file that was automatically created in the *Numbers* directory. Rename the *Class1.cs* file to *PrimeService.cs*.
30+
31+
```csharp
32+
namespace System.Numbers
33+
{
34+
public class PrimeService
35+
{
36+
public bool IsPrime(int candidate)
37+
{
38+
if (candidate < 2)
39+
{
40+
return false;
41+
}
42+
43+
for (int divisor = 2; divisor <= Math.Sqrt(candidate); ++divisor)
44+
{
45+
if (candidate % divisor == 0)
46+
{
47+
return false;
48+
}
49+
}
50+
return true;
51+
}
52+
}
53+
}
54+
```
55+
56+
> [!TIP]
57+
> It is worth mentioning the that `Numbers` class library was intentionally added to the `System` namespace. This allows for <xref:System.Math?displayProperty=fullName> to be accessible without a `using System;` namespace declaration. For more information, see [namespace (C# Reference)](../../csharp/language-reference/keywords/namespace.md).
58+
59+
### Create test projects
60+
61+
Create two new **xUnit Test Project (.NET Core)** templates from the same command prompt using the [`dotnet new xunit`](../tools/dotnet-new.md#test) command:
62+
63+
```dotnetcli
64+
dotnet new xunit -n XUnit.Coverlet.Collector
65+
```
66+
67+
```dotnetcli
68+
dotnet new xunit -n XUnit.Coverlet.MSBuild
69+
```
70+
71+
Both of the newly created xUnit test projects need to add a project reference of the *Numbers* class library. This is so that the test projects have access to the *PrimeService* for testing. From the command prompt, use the [`dotnet add`](../tools/dotnet-add-reference.md) command:
72+
73+
```dotnetcli
74+
dotnet add XUnit.Coverlet.Collector\XUnit.Coverlet.Collector.csproj reference Numbers\Numbers.csproj
75+
```
76+
77+
```dotnetcli
78+
dotnet add XUnit.Coverlet.MSBuild\XUnit.Coverlet.MSBuild.csproj reference Numbers\Numbers.csproj
79+
```
80+
81+
The *MSBuild* project is named appropriately, as it will depend on the [coverlet.msbuild](https://www.nuget.org/packages/coverlet.msbuild) NuGet package. Add this package dependency by running the [`dotnet add package`](../tools/dotnet-add-package.md) command:
82+
83+
```dotnetcli
84+
cd XUnit.Coverlet.MSBuild && dotnet add package coverlet.msbuild && cd ..
85+
```
86+
87+
The previous command changed directories effectively scoping to the *MSBuild* test project, then added the NuGet package. When that was done, it then changed directories, stepping up one level.
88+
89+
Open both of the *UnitTest1.cs* files, and replace their contents with the following snippet. Rename the *UnitTest1.cs* files to *PrimeServiceTests.cs*.
90+
91+
```csharp
92+
using System.Numbers;
93+
using Xunit;
94+
95+
namespace XUnit.Coverlet
96+
{
97+
public class PrimeServiceTests
98+
{
99+
readonly PrimeService _primeService;
100+
101+
public PrimeServiceTests() => _primeService = new PrimeService();
102+
103+
[
104+
Theory,
105+
InlineData(-1), InlineData(0), InlineData(1)
106+
]
107+
public void IsPrime_ValuesLessThan2_ReturnFalse(int value) =>
108+
Assert.False(_primeService.IsPrime(value), $"{value} should not be prime");
109+
110+
[
111+
Theory,
112+
InlineData(2), InlineData(3), InlineData(5), InlineData(7)
113+
]
114+
public void IsPrime_PrimesLessThan10_ReturnTrue(int value) =>
115+
Assert.True(_primeService.IsPrime(value), $"{value} should be prime");
116+
117+
[
118+
Theory,
119+
InlineData(4), InlineData(6), InlineData(8), InlineData(9)
120+
]
121+
public void IsPrime_NonPrimesLessThan10_ReturnFalse(int value) =>
122+
Assert.False(_primeService.IsPrime(value), $"{value} should not be prime");
123+
}
124+
}
125+
```
126+
127+
### Create a solution
128+
129+
From the command prompt, create a new solution to encapsulate the class library and the two test projects. Using the [`dotnet sln`](../tools/dotnet-sln.md) command:
130+
131+
```dotnetcli
132+
dotnet new sln -n XUnit.Coverage
133+
```
134+
135+
This will create a new solution file name `XUnit.Coverage` in the *UnitTestingCodeCoverage* directory. Add the projects to the root of the solution.
136+
137+
## [Linux](#tab/linux)
138+
139+
```dotnetcli
140+
dotnet sln XUnit.Coverage.sln add **/*.csproj --in-root
141+
```
142+
143+
## [Windows](#tab/windows)
144+
145+
```dotnetcli
146+
dotnet sln XUnit.Coverage.sln add (ls **/*.csproj) --in-root
147+
```
148+
149+
---
150+
151+
Build the solution using the [`dotnet build`](../tools/dotnet-build.md) command:
152+
153+
```dotnetcli
154+
dotnet build
155+
```
156+
157+
If the build is successful, you've created the three projects, appropriately referenced projects and packages, and updated the source code correctly. Well done!
158+
159+
## Tooling
160+
161+
There are two types of code coverage tools:
162+
163+
- **DataCollectors:** DataCollectors monitor test execution and collect information about test runs. They report the collected information in various output formats, such as XML and JSON. For more information, see [your first DataCollector](https://github.com/Microsoft/vstest-docs/blob/master/docs/extensions/datacollector.md).
164+
- **Report generators:** Use data collected from test runs to generate reports, often as styled HTML.
165+
166+
In this section, the focus is on data collector tools. To use Coverlet for code coverage, an existing unit test project must have the appropriate package dependencies, or alternatively rely on [.NET global tooling](../tools/global-tools.md) and the corresponding [coverlet.console](https://www.nuget.org/packages/coverlet.console) NuGet package.
167+
168+
## Integrate with .NET test
169+
170+
The xUnit test project template already integrates with [coverlet.collector](https://www.nuget.org/packages/coverlet.collector) by default.
171+
From the command prompt, change directories to the *XUnit.Coverlet.Collector* project, and run the [`dotnet test`](../tools/dotnet-test.md) command:
172+
173+
```dotnetcli
174+
cd XUnit.Coverlet.Collector && dotnet test --collect:"XPlat Code Coverage"
175+
```
176+
177+
> [!NOTE]
178+
> The `"XPlat Code Coverage"` argument is a friendly name that corresponds to the data collectors from Coverlet. This name is required but is case insensitive.
179+
180+
As part of the `dotnet test` run, a resulting *coverage.cobertura.xml* file is output to the *TestResults* directory. The XML file contains the results. This is a cross platform option that relies on the .NET Core CLI, and it is great for build systems where MSBuild is not available.
181+
182+
Below is the example *coverage.cobertura.xml* file.
183+
184+
```xml
185+
<?xml version="1.0" encoding="utf-8"?>
186+
<coverage line-rate="1" branch-rate="1" version="1.9" timestamp="1592248008"
187+
lines-covered="12" lines-valid="12" branches-covered="6" branches-valid="6">
188+
<sources>
189+
<source>C:\</source>
190+
</sources>
191+
<packages>
192+
<package name="Numbers" line-rate="1" branch-rate="1" complexity="6">
193+
<classes>
194+
<class name="Numbers.PrimeService" line-rate="1" branch-rate="1" complexity="6"
195+
filename="Numbers\PrimeService.cs">
196+
<methods>
197+
<method name="IsPrime" signature="(System.Int32)" line-rate="1"
198+
branch-rate="1" complexity="6">
199+
<lines>
200+
<line number="8" hits="11" branch="False" />
201+
<line number="9" hits="11" branch="True" condition-coverage="100% (2/2)">
202+
<conditions>
203+
<condition number="7" type="jump" coverage="100%" />
204+
</conditions>
205+
</line>
206+
<line number="10" hits="3" branch="False" />
207+
<line number="11" hits="3" branch="False" />
208+
<line number="14" hits="22" branch="True" condition-coverage="100% (2/2)">
209+
<conditions>
210+
<condition number="57" type="jump" coverage="100%" />
211+
</conditions>
212+
</line>
213+
<line number="15" hits="7" branch="False" />
214+
<line number="16" hits="7" branch="True" condition-coverage="100% (2/2)">
215+
<conditions>
216+
<condition number="27" type="jump" coverage="100%" />
217+
</conditions>
218+
</line>
219+
<line number="17" hits="4" branch="False" />
220+
<line number="18" hits="4" branch="False" />
221+
<line number="20" hits="3" branch="False" />
222+
<line number="21" hits="4" branch="False" />
223+
<line number="23" hits="11" branch="False" />
224+
</lines>
225+
</method>
226+
</methods>
227+
<lines>
228+
<line number="8" hits="11" branch="False" />
229+
<line number="9" hits="11" branch="True" condition-coverage="100% (2/2)">
230+
<conditions>
231+
<condition number="7" type="jump" coverage="100%" />
232+
</conditions>
233+
</line>
234+
<line number="10" hits="3" branch="False" />
235+
<line number="11" hits="3" branch="False" />
236+
<line number="14" hits="22" branch="True" condition-coverage="100% (2/2)">
237+
<conditions>
238+
<condition number="57" type="jump" coverage="100%" />
239+
</conditions>
240+
</line>
241+
<line number="15" hits="7" branch="False" />
242+
<line number="16" hits="7" branch="True" condition-coverage="100% (2/2)">
243+
<conditions>
244+
<condition number="27" type="jump" coverage="100%" />
245+
</conditions>
246+
</line>
247+
<line number="17" hits="4" branch="False" />
248+
<line number="18" hits="4" branch="False" />
249+
<line number="20" hits="3" branch="False" />
250+
<line number="21" hits="4" branch="False" />
251+
<line number="23" hits="11" branch="False" />
252+
</lines>
253+
</class>
254+
</classes>
255+
</package>
256+
</packages>
257+
</coverage>
258+
```
259+
260+
> [!TIP]
261+
> As an alternative, you could use the MSBuild package if your build system already makes use of MSBuild. From the command prompt, change directories to the *XUnit.Coverlet.MSBuild* project, and run the `dotnet test` command:
262+
>
263+
> ```dotnetcli
264+
> dotnet test --collect:"XPlat Code Coverage"
265+
> ```
266+
>
267+
> The resulting *coverage.cobertura.xml* file is output.
268+
269+
## Generate reports
270+
271+
Now that you're able to collect data from unit test runs, you can generate reports using [ReportGenerator](https://github.com/danielpalme/ReportGenerator). To install the [ReportGenerator](https://www.nuget.org/packages/dotnet-reportgenerator-globaltool) NuGet package as a [.NET global tool](../tools/global-tools.md), use the [`dotnet tool install`](../tools/dotnet-tool-install.md) command:
272+
273+
```dotnetcli
274+
dotnet tool install -g dotnet-reportgenerator-globaltool
275+
```
276+
277+
Run the tool and provide the desired options, given the output *coverage.cobertura.xml* file from the previous test run.
278+
279+
```console
280+
reportgenerator
281+
"-reports:Path\To\TestProject\TestResults\{guid}\coverage.cobertura.xml"
282+
"-targetdir:coveragereport"
283+
-reporttypes:Html
284+
```
285+
286+
After running this command, an HTML file represents the generated report.
287+
288+
:::image type="content" source="media/test-report.png" lightbox="media/test-report.png" alt-text="Unit test-generated report":::
289+
290+
## See also
291+
292+
- [Visual Studio unit test cover coverage](/visualstudio/test/using-code-coverage-to-determine-how-much-code-is-being-tested)
293+
- [GitHub - Coverlet repository](https://github.com/coverlet-coverage/coverlet)
294+
- [GitHub - ReportGenerator repository](https://github.com/danielpalme/ReportGenerator)
295+
- [ReportGenerator project site](https://danielpalme.github.io/ReportGenerator)
296+
- [.NET Core CLI test command](../tools/dotnet-test.md)
297+
298+
## Next Steps
299+
300+
> [!div class="nextstepaction"]
301+
> [Unit testing best practices](unit-testing-best-practices.md)

‎docs/core/toc.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -431,6 +431,8 @@ items:
431431
href: testing/selective-unit-tests.md
432432
- name: Order unit tests
433433
href: testing/order-unit-tests.md
434+
- name: Unit test code coverage
435+
href: testing/unit-testing-code-coverage.md
434436
- name: Unit test published output
435437
href: testing/unit-testing-published-output.md
436438
- name: Live unit test .NET Core projects with Visual Studio

0 commit comments

Comments
 (0)
⚡