The Cargo Cult of Test Driven Development
Wednesday, June 25, 2008 at 10:07PM Test Driven Development(TDD) is an agile practice that has been slowly gaining traction. In the last year I have encountered quite a few podcasts and blog posts that touted TDD as a better way to develop high quality software. I try to stay on top of trends in software development, and I really did attempt to use TDD, but in the end gave up as it just didn’t mesh with my own personal style.
I know many folks will say that you really need to practice TDD for 3 months to a year to start reaping the benefits, as was mentioned in this interesting discussion on the server side. That to me is a cop out. To me it is kind of like saying caviar is an acquired taste, when truthfully it just tastes like garbage.
For me, there is a big disconnect between everything you read on other blogs and podcasts about TDD and what you experience when actually trying to apply TDD in real life. I think it has a lot to do with the fact that proponents of TDD compare it to writing software without testing at all, as opposed to comparing it to a more structured approach that uses tests when they make sense.
After thinking through this and other issues that I touch in this blog post, I have come to the conclusion that TDD is in fact a cargo cult.
According to Wikipedia, Cargo Cult Programming refers to over-applying a design principle blindly without understanding the reasons behind that design principle in the first place. An example would be a novice being taught that commenting code is good, and then adding comments for lines that are self-explanatory or need no comment.
If you replaced “comments” in that last example with “unit tests”, you can begin to see how test driven development is a cargo cult.
Before we get in too deep, let me state empathetically that I am not against unit testing. They are an essential tool in a developer’s tool box. But when testing becomes dogma, that’s when I start to have problems.
Test First? How about Design First?
My biggest problem with TDD is that if feels like “just in time design.” Write a test to fulfill a single use case and design your api to satisfy one part of a use case. When the next use case has different requirements of your objects, modify your api and refactor. Repeat 1,000 times until you have high quality software.
I find refactoring in this fashion incredibly draining. To me, it is a lot of work, with very little pay off. I know there are plenty of tools that make refactoring easier, but as the complexity of your project grows, there is still a lot of pain in refactoring your code base. Don’t get me wrong, refactoring is a necessary part of any development process, especially when it will improve the maintainability of your codebase. I just don’t find it a terribly exciting task.
The other big problem I have with this approach is that when developing using TDD, you need to repeatedly switch contexts between coding and designing. As Joel Spolsky pointed out, context switching is considered harmful. I find that my frame of mind when designing is different than when I am coding. Having to constantly switch back and forth between designing and coding hampers my ability to get into the “zone:” that awesome place where productivity increases exponentially.
I find that if I spent time upfront really thinking about what the interfaces should look like and how they interact and behave before writing a line of code, the entire process flows better. At the very least, hashing out a class diagram and really trying to understand the responsibilities and collaborators (via CRC cards) of the objects you are building really provides value. Spending that kind of time up front also helps you see the kind of patterns that might bring additional strategic value to your design. When you are doing pure TDD, you are more in a tactical mind set, focused on solving your use cases and getting your tests to pass.
Everything Doesn’t Need to Have its Own Unit Test
I will admit that if you are looking to attain 100% code coverage, TDD is probably the way to go. But what does 100% code coverage actually buy you?
It doesn’t actually prove that your software is bug free. Testing every line of your code and every execution pathway are two entirely separate things. Test Quality is infinitely more useful than Test Quantity.
Also no amount of unit testing will ever replace user acceptance testing. Just because you wrote your code to spec, doesn’t mean that the spec didn’t make any faulty assumptions.
Another fact of life is that not all dependencies can be tested. Maybe you don’t know or can’t find out all the ways in which a dependency will behave. Maybe your class has a dependency on an object that can not be instantiated outside the environment it lives. SharePoint and other Web API’s come to mind. And while I am aware of Mock object frameworks, there gets to be a point where the effort required to mock an object is not worth the value it will bring from both an efficiency and quality point of view.
So why aim for 100% code coverage? I have read plenty of blog posts that say the difference between 99% code coverage and 100% coverage is immense. Some people will even test simple property getters and setters. If this isn’t overkill, I don’t know what is. To me this epitomizes cargo cult thinking.
When I develop software, I write plenty of unit tests. I don’t write them first, but I also don’t write them last. I write them as I write code complex enough that it warrants one. I also don’t just sit back to run unit tests and watch them pass or fail. I actively set breakpoints and debug them. I don’t always trust that a test that passes a test is actually functioning properly unless I step through it line by line. There are plenty of tools that integrate with your IDE to make this easy.
Debugging is something that I actually enjoy. Stepping through code, using locals and watch windows make developing fun. Why let the testing framework have all the fun.
No Silver Bullets for Software Quality
Writing Software is Hard and I am by no means an expert in anything, let alone TDD. If you find that TDD makes it easier for you to write software, by all means adopt it.
But I am fairly convinced that it is not a silver bullet that can be used on all projects and all situations. As it is with everything, the real skill is in identifying where and when it makes sense to use TDD and not trying to fit a round peg into a square hole.
If you agree or disagree with anything I said, leave a comment or meet me July 1st 2008 at the Fairfield/Westchester Dot Net user group meeting where I am participating in a panel discussion on testing.
Reader Comments (8)
That's not to say that there is no place for more formal design. Tdd & youre existing regression tests make that overhead unnecessary for a more complex class of tasks. There are still difficult architectural issues that need to be addressed with meetings, CDC cards, diagrams, etc.
I find the just in time design to be very effective, especially if your development iterations are very fast. TDD supports fast iterations. This offers more flexibility and lessens the cost of bduf screw ups (i.e. when its not perfect), which happens way more often than not, since its difficult to predict the future.
I also test the validity of my tests by using preconditions in the test and red/green/refactor. I find that these guards help the tests be appropriately fragile in the long run.
>> actually enjoy.
>> Stepping through code, using locals and
>> watch windows make developing fun.
>> Why let the testing framework have all the fun.
Becouse the time they spend doing it is CPU time, and is far cheaper then your time.
We always use TDD, but it isn't the only thing we do when our fingers are on the keyboard. Sure, I have to debug sometimes, but with the full knowledge that it's the most inefficient use of my time as a developer.
We also do _plenty_ of whiteboarding sessions to get the design in our heads. But the real design is the code, and the real verification of our design is in our tests.
I wouldn't get too hung up with folks touting "pure" TDD. 100% coverage is pointless; coverage is a metric, not a goal. TDD helps me deliver continuous, sustainable value, which is the whole point, right?
First, thanks for writing such a well reasoned article without resorting to invective. I think all the points you bring up are valid, and are more likely to spark productive conversation when presented in such a reasonable style.
While I am still a noob to TDD, I don't think I've ever heard any authority on TDD promote the idea of 100% test coverage. Is this a straw man argument, or have you actually heard this idea promoted?
I think it is important to remember that the test framework is being used for a purpose it was not specifically designed for. These are not tests in the QA sense, but a test framework is a convenient tool to enable this style of development. I am also not aware of any authority promoting the tests produced through TDD as a replacement for user acceptance tests.
Your point is valid that it is a fools errand to implement a complex system without some sort of rode map. TDD does not eliminate the role of an architect, but it does recognize the limitations of that role. Most development involves some element of design. The key benefit of TDD to me, at this point, is to make explicit when I am making a decision about what, as apart from how. I answer the what questions while writing the test, and the how questions during the implementation.
Regards,
++Alan
Too many people seem to be bogged down in 'doing things correctly' or what they perceive as correct.
Whenever I write an automate test I ask myself if I am really adding value. The only time I came close to 100% coverage was with framework code used for multiple teams, in that case it was worth me spending the extra time to ensure the code is 100% stable - as far as know no one has found a fault in that code yet! For most tasks we can tolerate some errors because the cost to fix is far cheaper than the cost/time required to achieve 100% code coverage.
Strict TDD practitioners should ask themselves if TDD makes sense in every case or are they wasting their clients time and money on as you say a Cargo Cult? I have used TDD in cases where little designed was required, but in general lean towards 'test-after' which is more efficient and pragmatic.
This view of context switching is the exact opposite of that held by Martin Fowler in his afterword to Kent Beck's Test Driven Development by Example (http://www.amazon.com/Test-Driven-Development-Addison-Wesley-Signature/dp/0321146530/ref=pd_bbs_1?ie=UTF8&s=books&qid=1215244130&sr=8-1). Here is the gist of Fowler's idea (I have turned excerpts of his afterword into bullet points):
* Programming is hard.
* It sometimes feels like trying to keep several balls in the air at once: any lapse of concentration and everything comes tumbling down.
* TDD helps reduce this feeling, and results in rapid unhurriedness (really fast progress despite feeling unhurried).
This is because working in a TDD development style gives you the sense of keeping just one ball in the air at once, so you can concentrate on that ball properly and do a really good job with it.
* When you are trying to add some new functionality, you are not worried about what really makes a good design for this piece of function, you are just trying to get a test to pass as easily as possible.
* When you switch to refactoring mode, you are not worried about adding some new function, you are just worried about getting the right design.
* With both of these activities, you are just focused on one thing at a time, and as a result you can concentrate better on that one thing.
* Adding features test-first and refactoring, are two monological flavours of programming.
* A large part of making activities systematic is identifying core tasks and allowing us to concentrate on only one at a time.
* An assembly line is a mind-numbing example of this - mind numbing because you always do the one thing.
* Perhaps what test-driven development suggests is a way of breaking apart the act of programming into elemental modes, but avoiding the monotony by switching rapidly between those modes.
* The combination of monological modes and switching gives you the benefits of focus and lowers the stress on the brain without the monotony of the assembly line.
We're about to embark on a huge web project using limited resources. My concern is interdependencies that often wouldn't be caught with Unit testing. I'd love to write automated assembly/system tests to reduce the # of man-hours spent regression testing, but didn't know if NUnit (or similar) was the right way to go.
If you have any thoughts, could you also forward your response to my e-mail (just in case I forget to check the comment thread).
Thanks for such a great post!
I also agree that there is no silver bullet to software development, and TDD will not solve all you problems.
But I think TDD is a very important development practice. And like Brian Takita said above, it also helps me to stay in the flow.
http://www.codeinstructions.com