Here am I introducing my Test-Driven Development and Code Coverage session for IMTC 2008:
Sign up here: http://imtc.firstport.ie/book.aspx
The IMTC 2008, is an 8 track, 40 session extravaganza covering a breadth of Microsoft’s latest emerging technologies. Throughout the festival there will be numerous networking opportunities, coffee and session re-runs.
Now in its third year, the IMTC 2008 is the second of seven major technology conferences in IrishDev.coms IxTC Series 2008. Co-organised with the Irish Microsoft Technology User Group and First Port Jobs, it’s an event by the Irish technology community for the Irish technology community.
The IMTC 2008 is being run over one evening and two action packed days – this is yet another Irish tech conference you cannot afford to miss!
Begins 7.00pm Wednesday 2nd April with Keynotes, 40 technology sessions on Thursday and Friday 3rd and 4th, concluding with post conference drinks.
It’s not free, but at €189 for two days and an evening of technical content, it’s excellent value!
I’ll be speaking at this event – Test-Driven Development and Code Coverage using Visual Studio 2008 Professional.
I’m demonstrating a specific product, so you might believe that it’s a “product plug” session that’s full of marketing stuff. Thankfully, nothing could be further from the truth. I’m an avid believer in making things simple – FinalBuilder is one of many products that help me achieve that aim. Therefore, I am demonstrating a useful and highly configurable tool.
FinalBuilder has built-in support for running NUnit tests, so it’s actually remarkably easy to include running tests in your automated build process. However, since FinalBuilder is so feature-rich, I wanted to demonstrate just how easy it is to write some FinalBuilder Actions that run the tests and stop the build if the tests fail. The key take-away from this post is the ease in which FinalBuilder can be customised to incorporate new and or as yet unsupported third party tools.
NUnit has two modes of operation: via a GUI or via the command-line console. Obviously the GUI provides nice visual feedback, red and green bars, etc. The console version is less visually pleasing, but does appeal to the command-line fraternity (which suits me!) Using the command-line version of NUnit (typically found here: C:\Program Files\NUnit 2.4.3\bin\Nunit-Console.exe), you’ll be pleased to know that it will run your tests on your behalf and create as output, create a nicely formatted XML document. That XML document contains two rather useful attributes: total and failures – these indicate the number of tests that were run and the number of failures.
Clearly we can make use of the failures attribute to our advantage. If it’s zero, then the automated build process can continue on. However, if it has a value of one or more, clearly we have a problem, the build is broken.
Without using FinalBuilder’s built-in NUnit Action, how might we go about incorporating NUnit into our FinalBuilder build process? Thanks to the power of FinalBuilder, it’s actually remarkably easy. Assuming that you have a new, clean FinalBuilder project, here’s what you do:
1. Goto the Tools -> Edit Variables menu, add a new variable called TestFailures.
2. Add a new Execute Program action (from the Windows OS action group). Set the Program File input box to point to nunit-console.exe. In the Parameters input box, enter the name of the DLL that contains your NUnit tests. In the Start In input box, enter the full path to the directory where the DLL that contains your tests can be found.
3. Add a new Define XML Document action (from the XML action group). Call the XML document TestResults. Set the Load document from file input box – set it to the TestResults.xml file that sits alongside the DLL that contains your tests. This assumes that you have either places an empty TestResults.xml file in that folder or you have run your tests through the NUnit Console prior to this exercise.
4. Add a new Read XML Value to Variable action. Set the XPath to Node equal to //test-results. Put a tick in the Read attribute check-box, set it equal to failures. From the Variable to Set drop-down menu, set it to TestFailures.
5. Add an If..Then action (from the Flow Control action group). Set the Left-hand Term equal to %TestFailures% – there is code completion to help you. Set the operator equal to “greater than”, i.e. >
6. Run your FinalBuilder project. If all goes well, i.e. the tests pass, the screenshots below should look familiar. Otherwise, if the tests fail, the whole build process fails.
This short example demonstrates the power of FinalBuilder – whilst there is a built-in action for running NUnit projects, this example has served to demonstrate how easy it is to integrate a third party tool into the FinalBuilder build process. Hopefully this short example has been enough to convince you that FinalBuilder can be used to integrate virtually any “build activity” that you may have in your process.
How are you carrying out your build process at the moment? Is it automated? Harness the power of the fully automated build!
The article is aging a little, but it demonstrates the main points. I’m planning to spend some time bringing my legacy content up to date, I’ll post an entry here as and when they do become available – watch this feed!
Today saw me run my Test-Driven Development and Code Coverage session for the third time in as many months. Whilst the audience was fewer in number than usual, the interactivity made it a good session for me (and for the attendees from what I can gather). One thing that came out of this delivery was the need to examine user interface testing in more detail – something for 2007 I think.
The source code, crib notes and slide deck are available here.
Seen as Benjamin Mitchell’s MSN Messenger caption:
NAnt, NUnit, NCover, NWorries
There’s a great line up of speakers available to choose from – some of whom are travelling from further afield to be here! And it sees the return of some familiar speakers, such as Steve Scott.
I have submitted a session, Unit Testing and Code Coverage: Putting it all together, here’s the abstract:
With Test-Driven Development (TDD) now entering the mainstream via such tools as NUnit and more recently Visual Studio 2005 Team System (VSTS), you may be wondering how to “get more” from TDD. I believe that we can improve the quality of our application by using a combination of TDD and code coverage.
Code coverage, whereby we “track” how much of our code is covered during testing, is not new. Indeed, we can practice code coverage and TDD in isolation. However, applying what we know about code coverage against our unit tests allows us to move our applications to the next “quality” level:
no longer is it acceptable to have unit tests on their own, we must know how well written the unit tests are, how much of the classes under test are really being tested?
Over the course of 60 minutes I will introduce the benefits of code coverage using both manual methods and automated tools. I will briefly introduce TDD and will go on to demonstrate the benefits of using code coverage tools against your unit tests, i.e. how well do your tests exercise your classes/application?
Whilst I will be using Visual Studio 2005 and C#, I will discuss Visual Studio 2003 compatibility too. I will be looking at a handful of the code coverage tools that are available, both free and commercial.
If you would like to see this session, please vote for it! And if there’s anything you would like me to cover in this session, please feel free to send me an e-mail or leave a comment here!
How well have your tests exercised your code?
Over the course of this posting I plan to demonstrate a number of tools and topics that encompass “testing”. I’ll be looking at code coverage – how much of our code is “exercised” or used. I’ll be looking at tools that can help us with code coverage. I’ll focus on using NUnit for testing and will demonstrate how we can tie it into the code coverage activity. Finally, I’ll be looking how we can integrate all of this good stuff into the Visual Studio IDE…using TestDriven.NET.
With that in mind, I will go through the following:
These four topics will demonstrate two things. Firstly, the benefit of using a code coverage tool to help you learn more about your application and the way that it works. Secondly, how that added benefit of a set of unit tests coupled with a code coverage tool can yield increased levels of confidence in your application’s testing strategy. Of course, prior to code coverage tools being automated, the simplest form of code coverage was the debugger: even as recently as 1998 I recall laboriously slaving over the Delphi debugger whilst I was “testing” my application. It was worth the toil, the few additional bugs that came to light meant that the application had years of trouble free use. Now, with code coverage integrated into the IDE, and with unit tests sitting side-by-side with the application source code, the time required to run the tests and perform a code coverage analysis is so short, it can be done with every build.
TestDriven.NET provides an integration layer between a number of testing frameworks, such as NUnit, and the IDE. Before TestDriven.NET, we would write our tests, write some code, build everything and then we would fire up a separate application that would run the tests (e.g. the NUnit front-end). Generally this is/was fine as the benefits of test execution greatly outweighed the use of a secondary application. Whilst the NUnit front-end allows us to choose which tests we want to run (as opposed to running all tests), we still find ourselves leaving the IDE and jumping into another application.
So, in addition to integrating NUnit into the Visual Studio IDE, it also provides integration with NCover and NCoverExplorer.
For the purposes of this posting, I am going to do and explain things in a slightly “out of order” fashion. I know that I have mentioned test-driven development already, and by rights we should be developing our application in that fashion. However, I would like to introduce code coverage first. Luckily the example that I plan to use is so simple, it could almost be tested without the need for test-driven development. The example that I plan to use to demonstrate code coverage is that of a simple calculator – I could have been more original, I apologise for my banality!
The Calculator Class
Implementing a simple calculator isn’t rocket science, but since I need you to be at least familiar with the code layout, here’s the code that I am using:
Here’s a little front-end WinForms application that makes use of [some] the MyCalculator class.
And here’s the code behind the WinForms application and the “add” button:
So, having compiled CalcApp, we can submit the .exe to NCover. Before we do that, it’s probably useful if I introduce NCover.
What is NCover?
NCover is a command-line tool that scans your application’s code and records information about which lines of code were executed. NCover uses the “sequence points” that the compiler generates and stores in the .pdb file (debug information) to determine which lines of code are executed and how frequently. Without getting bogged down in detail, a sequence point is essentially a single program state or a line of code.
NCover’s command-line syntax is:
Usage: NCover /c
/c Command line to launch profiled application.
/a List of assemblies to profile. i.e. "MyAssembly1;MyAssembly2"
/v Enable verbose logging (show instrumented code)
After NCover has analysed your code, it creates two files: coverage.log and coverage.xml.. The .log file contains the events and messages that NCover creates as it analyses your code base. The .xml file is very it all happens: it contains NCover’s analysis of your code base. There is a third file, coverage.xsl. It’s an XSL stylesheet that takes coverage.xml as its input, allowing it to be displayed in a neat tabular fashion inside Internet Explorer.
Running CalcApp through NCover
C:\Program Files\NCover>NCOVER.CONSOLE "...\dev\VS2005\Test\DDG\CalcApp\CalcApp\bin\Debug\CalcApp.exe"
NCover.Console v1.5.4 - Code Coverage Analysis for .NET - http://ncover.org
Copyright (c) 2004-2005 Peter Waldschmidt
Coverage Xml: Coverage.Xml
Coverage Log: Coverage.Log
Waiting for profiled application to connect...
******************* Program Output *******************
***************** End Program Output *****************
Alternatively, with TestDriven.NET installed, there is the IDE menu integration:
It’s important to note that the Coverage menu option is context sensitive. Depending upon “where” you right click, say either on the CalcApp solution or the TestLibrary, NCover will be invoked for the item you right clicked on. We will see this difference emerge later on in this posting when we demonstrate the difference between code coverage for CalcApp vs. code coverage for our NUnit tests.
Now, you may recall that I only implemented the “add” calculation. This is deliberate as I need to use the missing calculations to demonstrate code coverage.
Coverage.xml is too large to reprint here and it’s not all that easy to read. Fortunately, NCover’s author realised this and created an XSL (stylesheet) that transforms the XML into something a) more readable and b) more useful. The screenshot below presents a snapshot of that output – notice the green bars and red bars, we’re clearly in the ‘testing’ domain now. The full NCover coverage.xml file can be viewed here.
From this report, we can easily see that we’ve only covered 25% of the MyCalculator class. However, and this is a key point, in order to get this far, we had to perform manual testing. We had run the application “through NCover” such that it could watch what was happening. We had to enter the number 38 and 4 and we had to move the mouse and click on the Add button. Whilst this is a better than stepping though code with the debugger, it’s not automated therefore it’s not repeatable.
NCover’s report looks great and it serves a good purpose. NCoverExplorer moves things to the next level, it takes the output from NCover and presents it graphically in a treeview with integrated code viewing:
NCoverExplorer uses colour-coding to highlight the status of code coverage. It is configurable as the screenshot below demonstrates:
Introducing unit tests with NUnit
Without going into too much detail about test-driven development and NUnit (further information can be found in the Resources section of this post), our next step is to prepare a new class that allows us to test the MyCalculator class. In reality, we would have written our tests before writing the MyCalculator class, we’re doing things in a slightly different order for the sake of this post.
So, using Visual Studio we simply add a new Class Library to our solution, add a reference to the ClassUnderTest namespace and we’re off. The following code demonstrates how we might use the NUnit testing framework to exercise MyCalculator. Whilst not an NUnit requirement, I prefer to name my test methods with the prefix “test”, other unit testing frameworks may vary. As you can see, we’re simply recreating the manual test that we performed using the desktop application and we’re still only testing the “add” method.
Inviting NUnit to run these tests, in the absence of TestDriven.NET, would mean leaving the IDE. However, with TestDriven.NET, we can right click on the TestLibrary and run the tests using NUnit. The screenshot below presents the output from the NUnit front-end. It clearly demonstrates that the test for addition succeeded and with that we gain the confidence that “everything’s working”. However, what it doesn’t tell us is the fact that we’ve missed out testing some 75% of the MyCalculator class. For that, we need to use NCover on our tests.
Here’s a screenshot of NCoverExplorer viewing NCover’s Coverage.xml after it has analysed the tests:
The key takeaway from this screenshot, and indeed this posting, is the fact that we have automated our test and our code coverage: we can see in a single screenshot how well our tests are exercising our code.
I discovered NCover and NCoverExplorer via a couple of blog posts and was suitably impressed – I am always on the look out for ways of ensuring that my applications are as well tested as they can be. After all, there is nothing worse than a stream of ‘phone calls from your users each complaining about a show stopping crash or feature that does not appear to work. With careful use of the tools mentioned above, we can get ensure that our applications are tested and that we have no code that is unused. Code that is unused is often the source of bugs or feature failures. In the past, without tests and without code coverage tools, we had to resort to using a debugger to test all paths through our code – that was a laborious process fraught with repetition and boredom.
My recommendation: install NCover, install NCoverExplorer, install NUnit, install TestDriven.NET.
Update: before I could post this entry, the MSDN posting had been removed, luckily I saved a copy elsewhere. Since I’ve just spent the last few minutes writing this, I’m posting it anyway. Here’s what’s there now:
Guidelines for Test-Driven Development
This topic is obsolete and has been removed from the MSDN documentation.
It would appear that the MSDN posting has hit a nerve in the TDD community, and on first impressions, rightly so. I’m about to embark on a couple of days travel, I’ll be taking some of the material surrounding this posting and rant with me to read, expect to see me follow it up later this month.
To pick on a few posting (on the off chance you’re not following this rant elsewhere): Scott Dockendorf steps up and denounces any involvement in the article over here. Agile guru Roy Osherove is very vocal about the article in his Microsoft fails miserably to explain or promote Test Driven Development in Team System posting. And Julian M Bucknall is all depressed about it over here.
On a separate note, and it’s perhaps why I associated this posting with my “Opinion” category (not that this counts for much, but I feel as if that category lets me write what I like and not get sued! That’ll be right!), as an MVP should I be seen to be criticising Microsoft? Ignoring the relationship that exists, if an organisation did something that was wrong (in the eyes of the majority), I would feel professionally obliged to point out the error in their ways. Even after nearly a decade in the same organisation where such an attitude is pretty much frowned upon, I still maintain this professional desire to see things done properly. However, again on first impressions, I see that the MSDN posting is rapidly becoming MSTDD, so at least the differentiation is there. Hopefully all the TDD newbies will catch up on TDD postings prior to embarking on what amounts to a shoe-horned flavour of TDD.
More when I return from my travels.
Here’s the original MSDN posting:
Visual Studio Team System
Guidelines for Test-Driven Development
If your software-development project uses test-driven development, or TDD, you can benefit from features of Visual Studio 2005, and in particular, features of Visual Studio 2005 Team System. These features include the unit test type of Team Edition for Testers, and especially the ability to generate unit tests automatically; automatic refactoring capabilities that are introduced in Visual Studio 2005, and the Class Designer tool.
The unit test support of Team Edition for Testers is particularly suited to TDD because these Team System testing tools can generate tests from a minimum of production code.
In your TDD project, you might want to follow these steps:
- Define the requirements of your application.
- Familiarize yourself with the feature areas of your application, and decide on a single feature, or the requirements of a feature, to work on.
- Make a list of tests that will verify the requirements. A complete list of tests for a particular feature area describes the requirements of that feature area unambiguously and completely.
- File work items for feature requirements and for the tests that need to be written.
- In Visual Studio, create a project of the type you want. Visual Studio supplies the initial production code in the form of files such as Class1.cs, Program.cs, and Form1.cs, depending on the project type.
- Define the interfaces and classes for your feature or requirement. You can add a minimum of code, just enough to compile. Consider using the Class Designer to follow this step. For more information, see Designing Classes and Types.
The traditional TDD process does not contain this step. Instead, it advises that you create tests first. This step is included here so that, while creating tests, you can take advantage of two features in Visual Studio 2005 Team System: the GUI design capabilities of the Class Designer, and the automatic test-generation capabilities of Team Edition for Testers.
- Generate tests from your interfaces and classes. For more information, see How to: Generate a Unit Test.
- Compare the tests that have been generated with the list of tests you prepared in step 3. Create any tests that are missing from the list you wrote in step 3. For more information, see How to: Author a Unit Test.
- Organize your tests into test lists. You can, for example, base your lists on test use, such as check-in tests, BVTs, and tests for a full-test pass; or on area, such as UI tests, business-logic tests, and data-tier tests. For more information, see How to: Organize Tests into Test Lists.
- Update the generated tests to make sure they exercise the code in ways that cover the requirements that you have defined.
- Run your tests. For more information, see How to: Run Selected Tests.
Verify that all your tests fail. If any test produces a result of Inconclusive, it means that you did not update the generated code to add the appropriate verification logic for that test. If any test produces a result of Passed, it means that you did not implement the test correctly; this is because you have not implemented your production code, so no tests should pass.
- Implement the interfaces and classes of your production code.
- Run your tests again. If a test still fails, update your production code to respond. Repeat until all the tests pass.
- Pick the next feature or requirement to work on and repeat these steps.
With reference to this, Visual Studio 2005 Professional doesn’t come with the unit testing support that its big brother Visual Studio Team System does…
Channel 9 do a good job of getting life inside Microsoft presented to the outside world…here’s another fine example.
Download and watch the video here.
The interview was littered with good advice, but I homed in on a couple:
developers don’t like two things
- interfering management, i.e. micro-management
- management not doing something when something abnormal happens
“Incremental development (Ed.with regard to TDD) only works when refactoring is part of the equation”, Ron Jefferies
It’s well worth the download, do take the time to watch it. Clarke sums David’s style up rather well: “David has a nice way of simplifying complex things”
The first thing that Charlie said summed XP up for me: “lots of little bits that I’d being doing, but brought together”
Why is testing/TDD hard? A quick poll of the attendees revealed
Charlie followed it up with: testing connected sets of objects is hard because:
We went on to look at why GUI testing is hard.
GUIs are hard to test because there is tight coupling to other parts of the application, including the business logic. The solution to this is to separate the business logic from the UI, to use interfaces to permit substitution and factories to create business objects. This ran into an interesting discussion regarding the use of interfaces: some folks have argued that using interfaces isn’t the simplest and most expressive “thing”. Charlie believes that they are, on the grounds that they make it clear that a class has a capability. A similar discussion can be found here.
GUIs are hard to test because there is one large program handling all user interaction. Charlie’s recommendations revolved around using separate classes for UI logic and for validation. He also recommended the creation of smart controls (NUnit’s status bar being an example of such a thing – if you have NUnit installed, take a look here: C:\Program Files\NUnit 2.2\src\uikit). The key takeaway from this point was that the forms know too much about the objects that are dropped on them.
GUIs are hard to test because the design is driven by the IDE. Today’s modern IDEs will let you create a form, drop a button on it, double click on the button and write code for the button click. Granted, if you are doing the simplest thing, this is the right place for code, but in the long term it’s the wrong place for code (sure, you’d refactor it out, right?) We should resist the temptation to insert code into a UI just because the UI makes it easy. We should split code into two parts, presentation and behaviour (the use of partial classes and inheritance can help with this). Charlie recommend that we all read or re-read Mike Feather’s The Humble Dialog Box.
GUIs are hard to test because of technical issues of the platform. Charlie reiterated the need to be clear about what testing is needed, i.e. don’t test the platform (except to learn how the platform can help you with your testing), or put other way “don’t test other people’s stuff”
I liked Charlie’s “The Hardness Paradox” which basically stated that if something that is hard to test, it has to be a good thing. This will force us to look more deeply at how are applications are designed, with a view to making them easier to test. Humoursly, we compared this to a bang on the head – were it not for the fact that it hurts, we might keep doing it! Why should we shy away from difficult activities? We’ve cracked TDD for [elementary] classes, now we’re moving on to the next stage: testing the UI, the database, the web service, etc.
Quote for the day:
“If you don’t talk to the customer, the details don’t come up” – Charlie Poole, Edinburgh, 30th June 2005
(Oh, and NUnit 2.4 is under development! Can’t wait!)