Disclaimer

The views expressed on this weblog are mine alone and do not necessarily reflect the views of my employer, Avanade.

Search
Login
« Curb Your Enthusiasm with TDD | Main | The Cargo Cult of Test Driven Development »
Tuesday
Jul012008

Automated Testing with NUnit, MSTest, MbUnit and xUnit.Net

Although I am not a big fan of TDD, I am a strong proponent of automated testing. The majority of my experience has been using NUnit, although I have dabbled in Microsoft’s testing framework, MSTest from time to time. I have not used MSTest features that NUnit didn’t already have. Lately I have been hearing more about other unit testing frameworks like MbUnit and xUnit.Net and wanted to put together a high level comparison of the four frameworks to make it easier to decide which to use.

If you are familiar with any of these frameworks and need a quick reference to translate concepts from one framework to another, there is a nice comparison matrix on xUnit’s codeplex site.

The basic premise that most of these unit testing frameworks adhere to is you write a test that asserts some truth (or falsehood) about your code. Related tests are grouped into a test fixture, which allows you to add shared code that runs at the start (setup) and at the end (teardown) of your test execution. In most of the frameworks, test fixtures are classes and tests are methods.

Best practice is to create separate assemblies for your tests. So if you have a class library named “BusinessLogic,” you would create another assembly named “BusinessLogic.Tests” to house all the tests you will write. You can put tests in the same assembly as the classes themselves, but that would bug the heck out of me (it’s only a mild display of OCD).

In the beginning there was NUnit

NUnit was the first mainstream unit testing framework for .Net. It was originally ported from jUnit, a java unit testing framework which is a member of the xUnit (to be confused with xUnit.Net) family of frameworks that originated from SUnit, a Smalltalk unit testing framework.

So after you install the latest version of NUnit, you get a dll that you reference to access the NUnit framework and a graphical test tool which can be pointed to any assembly that has NUnit tests. Besides the core NUnit framework, you may want to install an additional tool like TestDriven.Net for tighter Visual Studio integration. Tools like this make it running and debugging tests trivial.

With NUnit, turning a class into a test fixture is as easy as adding a “[TestFixture]” attribute to your class and “[Test]” attribute to the methods that you want to test. A test will fail if an unexpected exception is thrown or an Assertion fails. Here is an example of a test that leverages NUnit from the NUnit Quickstart:

namespace bank
{

using System;
using NUnit.Framework;

[TestFixture]
public class AccountTest
{

Account source;
Account destination;

[SetUp]
public void Init()
{
source = new Account();
source.Deposit(200.00F);
destination = new Account();
destination.Deposit(150.00F);
}

[Test]
public void TransferFunds()
{
source.TransferFunds(destination, 100.00f);
Assert.AreEqual(250.00F, destination.Balance);
Assert.AreEqual(100.00F, source.Balance);
}

}

}

NUnit is probably the most mature unit testing framework of the bunch. It also has the biggest adoption and community support. There are more articles, discussions, tools and add ons for NUnit than any of the other frameworks combined. Just look at this link on google trends that compares searches for each of the frameworks. No other even comes close.

Reinventing the wheel with MSTest

MSTest made its first appearance with Visual Studio 2005, but only if you have the right sku. If you have VS.Net 2005 Professional or below, you were out of luck. You needed at least Enterprise or Team System Testing Edition to benefit from tight integration with VS.Net. Visual Studio 2008 gets it better, supporting MSTest in all flavors of VS.Net, including the Express editions.

The MSTest attributes are very similar to nUnit attributes. Instead of [TestFixture] you have [TestClass]. Instead of [Test] you have [TestMethod]. Instead of [Setup] you have [TestInitialize]. The Assert functions are even different than NUnit. If anything the MSTest versions are less intuitive than the NUnit attributes. Here is a sample test from a walkthrough I found (after a lot of searching) on msdn:

using Microsoft.VisualStudio.QualityTools.UnitTesting.Framework;
using System;

namespace VSTSDemo.Test
{

[TestClass()]
public class LogonInfoTest
{

[TestInitialize()]
public void Initialize()
{
// TODO: Add test initialization code
}

[TestCleanup()]
public void Cleanup()
{
// TODO: Add test cleanup code
}

[TestMethod]
public void ChangePasswordTest()
{
}

}
}

The main strength of MSTest is that it is bundled with Visual Studio. There will always be some shops out there that will only use stuff that comes from Microsoft, so it will always have a place. Besides that advantage, it also has the ability to generate test stubs for all the methods in a class or set of classes that you specify. This is a pretty neat trick, although its usefulness is somewhat limited in that it only creates one test per method, which is counter to what you typically need to do.

Microsoft’s own Patterns and Practices even acknowledge that there is no real compelling reason to move from NUnit to MSTest. For their flagship product, Enterprise Library, they provide unit tests that use both NUnit and MSTest framework by using compiler directives that allow you to build the tests for either framework.

Getting High with MbUnit

MbUnit, the only testing framework likened to a drug habit, builds on top of the syntax of NUnit, supporting all its behaviors and syntax as well as adding a bunch of useful additional capabilities that differentiate it. It also was built with extensibility in mind and exposes hooks that allow you to control every part of the process.

One of the cooler features of mbunit, is the concept of row tests. Row tests allow you to call a test multiple times with different parameters. Here is an example taken from mbUnit’s own tutorial:

[Row(3)]
[Row(6)]
[RowTest]
public void ToFizzBuzz_SendNumberDivisibleBy3ButNot5_ReturnsFizz(int NumberToTest)
{
Assert.AreEqual("fizz", FizzBuzz.FizzBuzz.ToFizzBuzz(NumberToTest));
}

MStest has a similar capability with the DataSource property, but it is nowhere as clean and requires you to put your data into a database, csv or xml file.

Migrating from NUnit to MbUnit is pretty straightforward, since you will only have to change assembly references, using statements and build scripts.

The only glaring weakness I see in MbUnit are in the documentation and tutorials. I know it has improved recently, but it is still weak in comparison to NUnit. There is also the likely possibility that some of the cooler features will be sucked back into NUnit. Someone already created a Rowtest extension for NUnit.

xUnit.Net - Just Got to be different

The main motivation behind xUnit.Net was to rethink unit testing based on past experiences. In the end, they came up with a new framework that has less than subtle differences with any of the other frameworks. Besides having a completely different syntax, xUnit.Net offers no setup or teardown methods (because they are evil) or expected exception attributes. They also aimed to take advantage of .Net 2.0 features like Generics which make the assertion syntax that much cleaner.

Comparing xUnit.Net to NUnit’s syntax, xUnit.Net has no [TestFixture] attribute, as test methods can be in any class. Instead of [Test] they have [Fact]. Instead of [Setup] you can use a parameterless constructor. Instead of Teardown you can implement IDisposable. They offer similar functionality to mbUnit’s [RowTest], but it is called [Theory]. Here is an example of a xUnit.Net Test from the codeplex tutorial.

public class MyTests
{

[Fact]
public void MyTest()
{
Assert.Equal(4, 2 + 2);
}

}

Besides the fact that xUnit is probably the least mature (current build is a release candidate),the biggest weakness for xUnit.Net is that it is so different from nUnit. Moving from nUnit to xUnit.Net does not look like a straight forward task. I still haven’t used xUnit.Net yet, but I do think some of the concepts, particularly around generics would be useful to have in a unit testing framework.

Wrapping Up

So after my shallow dive into these four unit testing frameworks, I think I need to look a little deeper into MbUnit. If you have any deeper experience with any of these frameworks and can enlighten me into any other advantages or disadvantages, please leave a comment.

EmailEmail Article to Friend

Reader Comments (5)

Thanks! This is the best survey and event tutorial like intro to automated testing I've seen. I love it.

--Paul Galvin
July 1, 2008 | Unregistered CommenterPaul Galvin
Being fair to Microsoft reinventing the wheel NUnit was not in widespread use back then. AFAIK Microsoft hired James Newkirk to head up the ms-test; previously he was the main developer on NUnit.

It is nice to see ms-test is now included in low spec versions of studio. It gives validity to Automated Testing in the eyes of pointy haired managers.
July 1, 2008 | Unregistered CommenterPaul Lockwood
Yes Microsoft did hire James Newkirk. MSTest is ok although a little slow.

I find the DataSource property invaluable. We don't use it, but it would potentially allow us (developers) to write the code and the testers throw in all the cases they like into the DataSource.

The easiest option would be to use csv's and have them in source control in a set folder so that QA only need to checkout that folder.

We're still using CruiseControl/Subversion/NUnit but have been looking at TFS. It's not perfect, but there are parts I like.

Web Tests, very easy to create and run under a number of browsers etc...

Load Tests, you can apply Web Tests, Unit Tests anything. It was quite interesting to Load Test some of our unit tests to see that the code also passes performance requirements.

Database Unit Tests, probably my favourite. It's very easy to populate (and randomise) dummy data etc... and tests sprocs for execution time, expected columns, expected result sets, all sorts.


Having said all that I think we'll be moving to MbUnit for typical unit tests. We can performance test with it, and it's easy enough to integrate into CControl.

We will be looking at Web Tests and Database Unit Tests though as part of TS.
August 16, 2008 | Unregistered CommenterRobert
I've wrote an extension for NUnit to experiment with data source tests (but that will be obsolete in the next version of NUnit), but also looked at theories. This blog post may be interesting:
http://taumuon-jabuka.blogspot.com/2008/03/comparing-theories-to-more-traditional.html

Cheers,
Gary
November 17, 2008 | Unregistered CommenterGary Evans
FYI while Jim Newkirk works at Microsoft he has never worked on MSTEST although he gave feedback to the team.

Also in VS2005 Unit Testing is available in Team Developer, Team Test and Team Suite.
In 2008 its available in the same SKUs and VS Pro (but not all features are in all SKUs). It is NOT available in Express
March 25, 2009 | Unregistered CommenterEuan Garden

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
All HTML will be escaped. Hyperlinks will be created for URLs automatically.