mtelligent

Although I am not a big fan of TDD, I am a strong proponent of automated testing. The majority of my experience has been using NUnit, although I have dabbled in Microsoft’s testing framework, MSTest from time to time. I have not used MSTest features that NUnit didn’t already have. Lately I have been hearing more about other unit testing frameworks like MbUnit and xUnit.Net and wanted to put together a high level comparison of the four frameworks to make it easier to decide which to use.

If you are familiar with any of these frameworks and need a quick reference to translate concepts from one framework to another, there is a nice comparison matrix on xUnit’s codeplex site.

The basic premise that most of these unit testing frameworks adhere to is you write a test that assets some truth (or falsehood) about your code. Related tests are grouped into a test fixture, which allows you to add shared code that runs at the start (setup) and at the end (teardown) of your test execution. In most of the frameworks, test fixtures are classes and tests are methods.

Best practice is to create separate assemblies for your tests. So if you have a class library named “BusinessLogic,” you would create another assembly named “BusinessLogic.Tests” to house all the tests you will write. You can put tests in the same assembly as the classes themselves, but that would bug the heck out of me (it’s only a mild display of OCD).

In the beginning there was NUnit

NUnit was the first mainstream unit testing framework for .Net. It was originally ported from jUnit, a java unit testing framework which is a member of the xUnit (to be confused with xUnit.Net) family of frameworks that originated from SUnit, a Smalltalk unit testing framework.

So after you install the latest version of NUnit, you get a dll that you reference to access the NUnit framework and a graphical test tool which can be pointed to any assembly that has NUnit tests. Besides the core NUnit framework, you may want to install an additional tool like TestDriven.Net for tighter Visual Studio integration. Tools like this make it running and debugging tests trivial.

With NUnit, turning a class into a test fixture is as easy as adding a “[TestFixture]” attribute to your class and “[Test]” attribute to the methods that you want to test. A test will fail if an unexpected exception is thrown or an Assertion fails. Here is an example of a test that leverages NUnit from the NUnit Quickstart:

namespace bank
{

using System;
using NUnit.Framework;

[TestFixture]
public class AccountTest
{

Account source;
Account destination;

[SetUp]
public void Init()
{
source = new Account();
source.Deposit(200.00F);
destination = new Account();
destination.Deposit(150.00F);
}

[Test]
public void TransferFunds()
{
source.TransferFunds(destination, 100.00f);
Assert.AreEqual(250.00F, destination.Balance);
Assert.AreEqual(100.00F, source.Balance);
}

}

}

NUnit is probably the most mature unit testing framework of the bunch. It also has the biggest adoption and community support. There are more articles, discussions, tools and add ons for NUnit than any of the other frameworks combined. Just look at this link on google trends that compares searches for each of the frameworks. No other even comes close.

Reinventing the wheel with MSTest

MSTest made its first appearance with Visual Studio 2005, but only if you have the right sku. If you have VS.Net 2005 Professional or below, you were out of luck. You needed at least Enterprise or Team System Testing Edition to benefit from tight integration with VS.Net. Visual Studio 2008 gets it better, supporting MSTest in all flavors of VS.Net, including the Express editions.

The MSTest attributes are very similar to nUnit attributes. Instead of [TestFixture] you have [TestClass]. Instead of [Test] you have [TestMethod]. Instead of [Setup] you have [TestInitialize]. The Assert functions are even different than NUnit. If anything the MSTest versions are less intuitive than the NUnit attributes. Here is a sample test from a walkthrough I found (after a lot of searching) on msdn:

using Microsoft.VisualStudio.QualityTools.UnitTesting.Framework;
using System;

namespace VSTSDemo.Test
{

[TestClass()]
public class LogonInfoTest
{

[TestInitialize()]
public void Initialize()
{
// TODO: Add test initialization code
}

[TestCleanup()]
public void Cleanup()
{
// TODO: Add test cleanup code
}

[TestMethod]
public void ChangePasswordTest()
{
}

}
}

The main strength of MSTest is that it is bundled with Visual Studio. There will always be some shops out there that will only use stuff that comes from Microsoft, so it will always have a place. Besides that advantage, it also has the ability to generate test stubs for all the methods in a class or set of classes that you specify. This is a pretty neat trick, although its usefulness is somewhat limited in that it only creates one test per method, which is counter to what you typically need to do.

Microsoft’s own Patterns and Practices even acknowledge that there is no real compelling reason to move from NUnit to MSTest. For their flagship product, Enterprise Library, they provide unit tests that use both NUnit and MSTest framework by using compiler directives that allow you to build the tests for either framework.

Getting High with MbUnit

MbUnit, the only testing framework likened to a drug habit, builds on top of the syntax of NUnit, supporting all its behaviors and syntax as well as adding a bunch of useful additional capabilities that differentiate it. It also was built with extensibility in mind and exposes hooks that allow you to control every part of the process.

One of the cooler features of mbunit, is the concept of row tests. Row tests allow you to call a test multiple times with different parameters. Here is an example taken from mbUnit’s own tutorial:

[Row(3)]
[Row(6)]
[RowTest]
public void ToFizzBuzz_SendNumberDivisibleBy3ButNot5_ReturnsFizz(int NumberToTest)
{
Assert.AreEqual("fizz", FizzBuzz.FizzBuzz.ToFizzBuzz(NumberToTest));
}

MStest has a similar capability with the DataSource property, but it is nowhere as clean and requires you to put your data into a database, csv or xml file.

Migrating from NUnit to MbUnit is pretty straightforward, since you will only have to change assembly references, using statements and build scripts.

The only glaring weakness I see in MbUnit are in the documentation and tutorials. I know it has improved recently, but it is still weak in comparison to NUnit. There is also the likely possibility that some of the cooler features will be sucked back into NUnit. Someone already created a Rowtest extension for NUnit.

xUnit.Net - Just Got to be different

The main motivation behind xUnit.Net was to rethink unit testing based on past experiences. In the end, they came up with a new framework that has less than subtle differences with any of the other frameworks. Besides having a completely different syntax, xUnit.Net offers no setup or teardown methods (because they are evil) or expected exception attributes. They also aimed to take advantage of .Net 2.0 features like Generics which make the assertion syntax that much cleaner.

Comparing xUnit.Net to NUnit’s syntax, xUnit.Net has no [TestFixture] attribute, as test methods can be in any class. Instead of [Test] they have [Fact]. Instead of [Setup] you can use a parameterless constructor. Instead of Teardown you can implement IDisposable. They offer similar functionality to mbUnit’s [RowTest], but it is called [Theory]. Here is an example of a xUnit.Net Test from the codeplex tutorial.

public class MyTests
{

[Fact]
public void MyTest()
{
Assert.Equal(4, 2 + 2);
}

}

Besides the fact that xUnit is probably the least mature (current build is a release candidate),the biggest weakness for xUnit.Net is that it is so different from nUnit. Moving from nUnit to xUnit.Net does not look like a straight forward task. I still haven’t used xUnit.Net yet, but I do think some of the concepts, particularly around generics would be useful to have in a unit testing framework.

Wrapping Up

So after my shallow dive into these four unit testing frameworks, I think I need to look a little deeper into MbUnit. If you have any deeper experience with any of these frameworks and can enlighten me into any other advantages or disadvantages, please leave a comment.

Test Driven Development(TDD) is an agile practice that has been slowly gaining traction. In the last year I have encountered quite a few podcasts and blog posts that touted TDD as a better way to develop high quality software. I try to stay on top of trends in software development, and I really did attempt to use TDD, but in the end gave up as it just didn’t mesh with my own personal style.

I know many folks will say that you really need to practice TDD for 3 months to a year to start reaping the benefits, as was mentioned in this interesting discussion on the server side. That to me is a cop out. To me it is kind of like saying caviar is an acquired taste, when truthfully it just tastes like garbage.

For me, there is a big disconnect between everything you read on other blogs and podcasts about TDD and what you experience when actually trying to apply TDD in real life. I think it has a lot to do with the fact that proponents of TDD compare it to writing software without testing at all, as opposed to comparing it to a more structured approach that uses tests when they make sense.

After thinking through this and other issues that I touch in this blog post, I have come to the conclusion that TDD is in fact a cargo cult.

According to Wikipedia, Cargo Cult Programming refers to over-applying a design principle blindly without understanding the reasons behind that design principle in the first place. An example would be a novice being taught that commenting code is good, and then adding comments for lines that are self-explanatory or need no comment.

If you replaced “comments” in that last example with “unit tests”, you can begin to see how test driven development is a cargo cult.

Before we get in too deep, let me state empathetically that I am not against unit testing. They are an essential tool in a developer’s tool box. But when testing becomes dogma, that’s when I start to have problems.

Test First? How about Design First?

My biggest problem with TDD is that if feels like “just in time design.” Write a test to fulfill a single use case and design your api to satisfy one part of a use case. When the next use case has different requirements of your objects, modify your api and refactor. Repeat 1,000 times until you have high quality software.

I find refactoring in this fashion incredibly draining. To me, it is a lot of work, with very little pay off. I know there are plenty of tools that make refactoring easier, but as the complexity of your project grows, there is still a lot of pain in refactoring your code base. Don’t get me wrong, refactoring is a necessary part of any development process, especially when it will improve the maintainability of your codebase. I just don’t find it a terribly exciting task.

The other big problem I have with this approach is that when developing using TDD, you need to repeatedly switch contexts between coding and designing. As Joel Spolsky pointed out, context switching is considered harmful.  I find that my frame of mind when designing is different than when I am coding. Having to constantly switch back and forth between designing and coding hampers my ability to get into the “zone:” that awesome place where productivity increases exponentially.

I find that if I spent time upfront really thinking about what the interfaces should look like and how they interact and behave before writing a line of code, the entire process flows better. At the very least, hashing out a class diagram and really trying to understand the responsibilities and collaborators (via CRC cards) of the objects you are building really provides value. Spending that kind of time up front also helps you see the kind of patterns that might bring additional strategic value to your design. When you are doing pure TDD, you are more in a tactical mind set, focused on solving your use cases and getting your tests to pass.

Everything Doesn’t Need to Have its Own Unit Test

I will admit that if you are looking to attain 100% code coverage, TDD is probably the way to go. But what does 100% code coverage actually buy you?

It doesn’t actually prove that your software is bug free. Testing every line of your code and every execution pathway are two entirely separate things. Test Quality is infinitely more useful than Test Quantity.

Also no amount of unit testing will ever replace user acceptance testing. Just because you wrote your code to spec, doesn’t mean that the spec didn’t make any faulty assumptions.

Another fact of life is that not all dependencies can be tested. Maybe you don’t know or can’t find out all the ways in which a dependency will behave. Maybe your class has a dependency on an object that can not be instantiated outside the environment it lives. SharePoint and other Web API’s come to mind. And while I am aware of Mock object frameworks, there gets to be a point where the effort required to mock an object is not worth the value it will bring from both an efficiency and quality point of view.

So why aim for 100% code coverage? I have read plenty of blog posts that say the difference between 99% code coverage and 100% coverage is immense. Some people will even test simple property getters and setters. If this isn’t overkill, I don’t know what is. To me this epitomizes cargo cult thinking.

When I develop software, I write plenty of unit tests. I don’t write them first, but I also don’t write them last. I write them as I write code complex enough that it warrants one. I also don’t just sit back to run unit tests and watch them pass or fail. I actively set breakpoints and debug them. I don’t always trust that a test that passes a test is actually functioning properly unless I step through it line by line. There are plenty of tools that integrate with your IDE to make this easy.

Debugging is something that I actually enjoy. Stepping through code, using locals and watch windows make developing fun. Why let the testing framework have all the fun.

No Silver Bullets for Software Quality

Writing Software is Hard and I am by no means an expert in anything, let alone TDD. If you find that TDD makes it easier for you to write software, by all means adopt it.

But I am fairly convinced that it is not a silver bullet that can be used on all projects and all situations. As it is with everything, the real skill is in identifying where and when it makes sense to use TDD and not trying to fit a round peg into a square hole.

If you agree or disagree with anything I said, leave a comment or meet me July 1^st 2008 at the Fairfield/Westchester Dot Net user group meeting where I am participating in a panel discussion on testing.

Yesterday, I took and passed exam 70-431. I am now a Microsoft Certified Technical Specialist in SQL Server 2005.

Since my last post, I reviewed the detailed notes for the test, which were in my opinion not too deep. The covered all of the topics, but not very deep and organized in a haphazard manor. 

I also took 3 lengthy practice exams, and one 40 question final exam. The questions on these test were high quality and indicative of the kinds of questions I saw on the exam. The explanations on the answers were good, but no where near the quality of the answers to the diagnostic exam I detailed in a previous post.

Some of the questions were somewhat interactive but no where near the interactivity of the simulations that were on the actual exam. This was actually the first exam I took that had simulations and they were pretty detailed and I definitely hope they expand on this approach in other tests.

After each practice exam, my score went up a little. I actually did well on the third practice exam. The prep kit had a nice little feature where it took all the questions I got wrong on previous tests, mixed it with a few I got right to keep me honest and let me retake everything. I kept taking this type of exam until I had successfully answered each question.

In the end, I didn't feel very good when I went into the exam. Normally I feel really prepared when going in. But then, I couldn't shake this fear that I hadn't been prepared enough, that I had really only prepared for the specific questions on the practice exam.

As I took the exam, many of the questions were pretty topical, given the material , but there were a few I wasn't prepared for. After I answered all the questions, I wasn't extremely confident that I had passed.

I declined to comment on any of the questions and held my breath for the two-three second eternity it took to calculate my score. I passed with a 741 out of 1000. Not a great score, but since 700 is passing, it would have to do.

After going through this process, I am still pretty happy about the uCertify prep kit, However I would not recommend it as the sole source of information when preparing for the exam. I think by taking a balanced approach of reading a detailed book, and using the uCertify prepKit practice questions would be a good formula for results.

So I have begun my journey for SQL Server exam 70-431 exam preparedness with a uCertify prepkit as my only guide. I have to admit that it has been a while since the knowledge that helped me attain my MCDBA in SQL Server 2000 flowed through my brain on a daily basis. And although I have used SQL Server 2005 extensively during development, I haven’t had to do as much work on the administrative side.

When you first open the prepkit after installation you are given a number of different options were to start. Your eye is immediately drawn to the first option to take a diagnostic test of 15 questions. Honestly I wasn’t sure what was even on the exam, so I was a bit hesitant to go that route. I instead chose to click the “Exam objectives”

I was a little surprised that the objectives didn’t jive directly with the objectives listed on the Microsoft web site, although the topics were generally in sync. There was a button to go to a more detailed view with exam notes that looked pretty extensive.

Once I felt comfortable with the topics the exam covered I decided to take the diagnostic test after all. The exam was 15 questions that covered a bunch of different topics on the exam. The questions were decent, but the most surprising thing was that not one of them was a multiple choice question. They were a lot of drag and drop answers in the right order type questions, click on the right place in the image type questions.

I actually think these types of questions are better that the standard multiple choice questions you typically get on a MCP exam. By better I mean I think they better demonstrate you know the material that make a random guess and happen to be right. The one caveat to that is that it might give you the impression that the actual MCP exam will have its questions structured like that, when in my experience at most you got a couple of questions like that with the rest being the standard multiple choice trickery.

I ended up getting 8 out of 15 questions correct. My basic knowledge of SQL Server got me that far and I knew I had some areas I would need to sharpen and deepen my knowledge of. The best part was actually after I finished the test.

The explanations for the correct answers were very detailed. Most all of them included multiple screen shots. A few actually had flash presentations that felt more like a mini training demonstration that a test question explanation.

The test interface gave you the option to tag a question, add notes, give feedback, and discuss the item. I tried to enter feedback for a question and am not really sure what happens to the feedback. I know it saves it, but after it gets sent, who knows. I also thought the discuss feature was pretty interesting. The idea of multiple test takers having discussions about a particular question seemed powerful, but after adding a comment, it was displayed with a note saying it wasn’t approved. None of the 15 questions had any comments tied to them, so I don’t know if this is because no one is commenting or no one is approving them. If it is working and used widely, I can see this being a pretty powerful feature.

Next I’ll be digging a little deeper into the articles, notes and other questions that the prep kit offers.

Stay tuned for more…

As most of you know, I am a bit of a Microsoft certification junkie. All in all I have passed 10 different exams and hold a whole bunch of credentials including MCP, MCSD.Net, MCAD, MCPD-EA, MCTS and the MCDBA for SQL Server 2000. The first exam I ever took was on configuring Windows 2000 Professional, back in 2001 before I made the decision to pursue development opportunities over systems administration.

After passing so many exams, I think I have been able to figure out a good formula for preparing for exams. I usually take the following approach:

• Cram, Cram, Cram – Buy a detailed exam book and read it front to back. I usually don’t pause to do any of the hands on labs unless it is dealing with something I am really not familiar with. I usually am able to blow through these books in a week or two.
  
• Test, Review and Test again – Besides the detailed exam book, I also typically try to get some practice tests to help reinforce the material. I avoid taking any practice exams until I at least have some familiarity with all the material. This is so I get the full benefit of researching the errors that I make and learning from my mistakes. I have tried many resources for this part including Exam Cram books, MSFT review books, Transcenders (paid for by my company, Avanade) and various sources on the .Net.

All in all this process has lead to me passing every exam I ever took on the first try. I did fail beta versions of exam 70-553 and 70-554, but only because there were absolutely no study materials available at the time I took the tests. That was the main reason I started this blog and posted all the notes I created for myself while studying for those betas.

Having completed the MCPD-EA, the next credential I wanted to update was my SQL Server certification. As I mentioned earlier, I already have a MCDBA in SQL Server 2000, but it is probably time for me to upgrade that to the MCTS or MCITP for SQL Server (Especially since 2008 exams are almost in beta). This goal has been on the back burner for a while since I have had my attention split between SharePoint, Silverlight and Snip-It Pro. (Snip-It Pro is a snippet productivity tool I built)

Recently, Roger Stuart from uCertify  contacted me about reviewing their certification prep kits. I thought this was the perfect excuse to move that goal to the front burner and kill two birds with one stone, reviewing their materials and preparing for one of the SQL Server 2005 exams.

As far as I can tell, a uCertify prepkit is more than just a bank of test questions. It includes a collection of articles, flash cards and study notes as well as diagnostic, practice and adaptive tests. To really test the effectiveness of their materials and I am going to forgo my typical preparation approach and try to use the uCertify prep kit exclusively to prepare for exam 70-431 – Microsoft SQL Server 2005 Implementation and Maintenance.

This is the first in a series of posts on what my impressions and experience is with the prep kit and it should culminate with me passing my 11^th exam.

Stay Tuned…