Automated Code Coverage and Unit Tests

How well have your tests exercised your code?

Over the course of this posting I plan to demonstrate a number of tools and topics that encompass “testing”. I’ll be looking at code coverage – how much of our code is “exercised” or used. I’ll be looking at tools that can help us with code coverage. I’ll focus on using NUnit for testing and will demonstrate how we can tie it into the code coverage activity. Finally, I’ll be looking how we can integrate all of this good stuff into the Visual Studio IDE…using TestDriven.NET.

With that in mind, I will go through the following:

  1. The creation of a simple class that does something trivial, in this case a calculator (Visual Studio, C#)
  2. The creation of a simple front-end application that uses the calculator (Visual Studio, C#)
  3. An examination of code coverage using the front-end application (NCover and NCoverExplorer)
  4. An examination of code coverage using unit tests (NUnit)

These four topics will demonstrate two things. Firstly, the benefit of using a code coverage tool to help you learn more about your application and the way that it works. Secondly, how that added benefit of a set of unit tests coupled with a code coverage tool can yield increased levels of confidence in your application’s testing strategy. Of course, prior to code coverage tools being automated, the simplest form of code coverage was the debugger: even as recently as 1998 I recall laboriously slaving over the Delphi debugger whilst I was “testing” my application. It was worth the toil, the few additional bugs that came to light meant that the application had years of trouble free use. Now, with code coverage integrated into the IDE, and with unit tests sitting side-by-side with the application source code, the time required to run the tests and perform a code coverage analysis is so short, it can be done with every build.

TestDriven.NET provides an integration layer between a number of testing frameworks, such as NUnit, and the IDE. Before TestDriven.NET, we would write our tests, write some code, build everything and then we would fire up a separate application that would run the tests (e.g. the NUnit front-end). Generally this is/was fine as the benefits of test execution greatly outweighed the use of a secondary application. Whilst the NUnit front-end allows us to choose which tests we want to run (as opposed to running all tests), we still find ourselves leaving the IDE and jumping into another application.

So, in addition to integrating NUnit into the Visual Studio IDE, it also provides integration with NCover and NCoverExplorer.

Code Coverage
For the purposes of this posting, I am going to do and explain things in a slightly “out of order” fashion. I know that I have mentioned test-driven development already, and by rights we should be developing our application in that fashion. However, I would like to introduce code coverage first. Luckily the example that I plan to use is so simple, it could almost be tested without the need for test-driven development. The example that I plan to use to demonstrate code coverage is that of a simple calculator – I could have been more original, I apologise for my banality!

The Calculator Class
Implementing a simple calculator isn’t rocket science, but since I need you to be at least familiar with the code layout, here’s the code that I am using:


Here’s a little front-end WinForms application that makes use of [some] the MyCalculator class.


And here’s the code behind the WinForms application and the “add” button:


So, having compiled CalcApp, we can submit the .exe to NCover. Before we do that, it’s probably useful if I introduce NCover.

What is NCover?
NCover is a command-line tool that scans your application’s code and records information about which lines of code were executed. NCover uses the “sequence points” that the compiler generates and stores in the .pdb file (debug information) to determine which lines of code are executed and how frequently. Without getting bogged down in detail, a sequence point is essentially a single program state or a line of code.

NCover’s command-line syntax is:

Usage: NCover /c [/a ]

/c Command line to launch profiled application.
/a List of assemblies to profile. i.e. "MyAssembly1;MyAssembly2"
/v Enable verbose logging (show instrumented code)

After NCover has analysed your code, it creates two files: coverage.log and coverage.xml.. The .log file contains the events and messages that NCover creates as it analyses your code base. The .xml file is very it all happens: it contains NCover’s analysis of your code base. There is a third file, coverage.xsl. It’s an XSL stylesheet that takes coverage.xml as its input, allowing it to be displayed in a neat tabular fashion inside Internet Explorer.

Running CalcApp through NCover
NCover’s output:

C:\Program Files\NCover>NCOVER.CONSOLE "...\dev\VS2005\Test\DDG\CalcApp\CalcApp\bin\Debug\CalcApp.exe"
NCover.Console v1.5.4 - Code Coverage Analysis for .NET -
Copyright (c) 2004-2005 Peter Waldschmidt

Command: ...\dev\VS2005\Test\DDG\Calc
Command Args:
Working Directory:
Coverage Xml: Coverage.Xml
Coverage Log: Coverage.Log

Waiting for profiled application to connect...
******************* Program Output *******************
***************** End Program Output *****************

Alternatively, with TestDriven.NET installed, there is the IDE menu integration:


It’s important to note that the Coverage menu option is context sensitive. Depending upon “where” you right click, say either on the CalcApp solution or the TestLibrary, NCover will be invoked for the item you right clicked on. We will see this difference emerge later on in this posting when we demonstrate the difference between code coverage for CalcApp vs. code coverage for our NUnit tests.

Now, you may recall that I only implemented the “add” calculation. This is deliberate as I need to use the missing calculations to demonstrate code coverage.

Coverage.xml is too large to reprint here and it’s not all that easy to read. Fortunately, NCover’s author realised this and created an XSL (stylesheet) that transforms the XML into something a) more readable and b) more useful. The screenshot below presents a snapshot of that output – notice the green bars and red bars, we’re clearly in the ‘testing’ domain now. The full NCover coverage.xml file can be viewed here.


From this report, we can easily see that we’ve only covered 25% of the MyCalculator class. However, and this is a key point, in order to get this far, we had to perform manual testing. We had run the application “through NCover” such that it could watch what was happening. We had to enter the number 38 and 4 and we had to move the mouse and click on the Add button. Whilst this is a better than stepping though code with the debugger, it’s not automated therefore it’s not repeatable.

NCover’s report looks great and it serves a good purpose. NCoverExplorer moves things to the next level, it takes the output from NCover and presents it graphically in a treeview with integrated code viewing:


NCoverExplorer uses colour-coding to highlight the status of code coverage. It is configurable as the screenshot below demonstrates:


Introducing unit tests with NUnit
Without going into too much detail about test-driven development and NUnit (further information can be found in the Resources section of this post), our next step is to prepare a new class that allows us to test the MyCalculator class. In reality, we would have written our tests before writing the MyCalculator class, we’re doing things in a slightly different order for the sake of this post.

So, using Visual Studio we simply add a new Class Library to our solution, add a reference to the ClassUnderTest namespace and we’re off. The following code demonstrates how we might use the NUnit testing framework to exercise MyCalculator. Whilst not an NUnit requirement, I prefer to name my test methods with the prefix “test”, other unit testing frameworks may vary. As you can see, we’re simply recreating the manual test that we performed using the desktop application and we’re still only testing the “add” method.


Inviting NUnit to run these tests, in the absence of TestDriven.NET, would mean leaving the IDE. However, with TestDriven.NET, we can right click on the TestLibrary and run the tests using NUnit. The screenshot below presents the output from the NUnit front-end. It clearly demonstrates that the test for addition succeeded and with that we gain the confidence that “everything’s working”. However, what it doesn’t tell us is the fact that we’ve missed out testing some 75% of the MyCalculator class. For that, we need to use NCover on our tests.


Here’s a screenshot of NCoverExplorer viewing NCover’s Coverage.xml after it has analysed the tests:


The key takeaway from this screenshot, and indeed this posting, is the fact that we have automated our test and our code coverage: we can see in a single screenshot how well our tests are exercising our code.

I discovered NCover and NCoverExplorer via a couple of blog posts and was suitably impressed – I am always on the look out for ways of ensuring that my applications are as well tested as they can be. After all, there is nothing worse than a stream of ‘phone calls from your users each complaining about a show stopping crash or feature that does not appear to work. With careful use of the tools mentioned above, we can get ensure that our applications are tested and that we have no code that is unused. Code that is unused is often the source of bugs or feature failures. In the past, without tests and without code coverage tools, we had to resort to using a debugger to test all paths through our code – that was a laborious process fraught with repetition and boredom.

My recommendation: install NCover, install NCoverExplorer, install NUnit, install TestDriven.NET.

Further Reading
TestDriven.NET Google Group
Microsoft Team System
Peter Waldschmidt.’s NCover
NCoverExplorer FAQ

DotNet Developers Group articles:
Test-Driven Development: a practical guide, Nov/Dec 2003
Test-Driven Development Using csUnit in C#Builder, Jan/Feb 2004

13 thoughts on “Automated Code Coverage and Unit Tests”

  1. Hey thanks for that, Craig!
    We were using csUnit with VS2003 and found it fantastic. Now we’re on NUnit with VS2005 (csUnit hasn’t moved in a good while now) and are a little disappointed. This is a much better level of integration and gives us a csUnit-like ‘feel-good’ factor again.
    So, thanks for that!

    It’s just a pity that our ASP.NET guy is still annoyed that NUnit can’t run tests on VS2005 web projects :(.

  2. Nice post. I still feel that NUnit and other open source tools offer a better experience for doing TDD than VS2005. The whole test list idea is horrible and they haven’t thought about things like the Category attribute. Another problem with VS2005 testing tools is poor support for inheritance structures. I’ve enjoyed this article, good job.

  3. David, Maruis – glad you liked the post, thanks for the positive feedback!

    Even with some heavy TDDers on the Microsoft team (James Newkirk, et al), I agree, the open source TDD toolset seems to be a nicer fit than the out of the box offering in Visual Studio 2005 – I’d like to think this situation will change over time though.

  4. Pingback: koido Blog
  5. Good post. Which version of VS2005 are you using?

    The Test edition, gives you most of this goodness out of the box, and almost seamless integration. We had some challenges at my previous work with NCover and nunit, so we used CoverageEye instead to achieve similar results.


  6. I’m using Visual Studio 2005 Pro for this posting. I’ll take a look at CoverageEye – thanks for the heads up!

  7. Nice writeup. Thank you for putting this out there – it helped pull a few things together for me. Just a note for David above – There is some work on the ASP.Net front that will make your guy happy (maybe):

    Scott Guthrie has been posting about the Visual Studio Web Application Project which moves things back to the 2003 model of web developement. Some would say that is a good thing – it does enable testing again. Check his main blog postings for updates but the final version should be available soon.

    Scott Hanselman has a post about a nice way to test ASP.Net Pages NUnit Unit Testing of ASP.NET Pages, Base Classes, Controls and other widgetry using Cassini (ASP.NET Web Matrix/Visual Studio Web Developer)

    Phil Haack has a post about a lightweight version of the Hansleman method. Simulating Http Context For Unit Tests Without Using Cassinni nor IIS

    I am still getting the hang of all this but the wealth of concerned people out there posting articles like yours will make the difference.

    Thanks again.

  8. Any codecoverage tool that can profile on the fly.I mean without code modification or recompilation? Something like code profiler (e.g. ProfileSharp)

  9. Hi Thanks it was explained nicely what i needed. Have done a good job

  10. Nice work. To add, i wonder whether this coverage could be used to have coverage analysis of sourcecode of unmanaged code that gets called from the managed one. for eg. coverage details of an unmanaged .c file getting called from a .cpp test suite.

  11. Good overview, I think many people will find it helpful. We’ve written a plugin called TestMatrix which takes things a step further by integrated the coverage tracking right into Visual Studio. It allows you to see every line that was missed, right there in the editor. That way you aren’t jumping from tool to tool.

Comments are closed.