Is Test Driven Development Effective

Yesterday Phil Haack and Jacob Proffitt reviewed "On the Effectivenes of Test-first Approach to Programming" by National Research Canada and published in the Proceedings of the IEEE Transactions on Software Engineering, 31(1), January 2005.

The test-first approach is often called Test Driven Development, in that the tests are written first and then "drive" the development. This approach is often encouraged in agile programming environments.

Phil posted first. After explaining his own intuition that TDD is a beneficial approach, Pill cited various sections of the report, including the abstract, that support the benefits of the test-first or TDD method.

Jacob posted after reading Phil's analysis. After explaining his own scepticism with reports like this (including his suspicion of confirmation bias), Jacob cited several sections of the report that seemed to show the opposite of what Phil and even the report abtract claimed.

Jacob made some interesting observations:

• The control group (non-TDD or "Test Last") had higher quality in every dimension—they had higher floor, ceiling, mean, and median quality.
• The control group produced higher quality with consistently fewer tests.
• Quality was better correlated to number of tests for the TDD group (an interesting point of differentiation that I'm not sure the authors caught).
• The control group's productivity was highly predictable as a function of number of tests and had a stronger correlation than the TDD group.

I have experimented with "test first", but found it frustrating. I always ended up just "faking" some tests so I could get on with the programming (Git 'Er Done). Then writing tests after to confirm that I met the requirements. This seems to correlate with Jacob's experience.

Both Jacob and Phil end their analyses with a call for more research. Me too.

This site looks much better in a browser that supports current web standards, but it is accessible to any browser. Download one now