Better Software - December 2007 - (Page 35)

Another customer comes along with a killer bug. The development team goes in and tries to find it, but the system is doing exactly what it is supposed to be doing; all the tests pass, but the system doesn’t work right. After three days of head scratching, the team finds that one of the tests is wrong—just a coding error in the Java code for the test. Oops. Once the test is fixed, it fails. It turns out that this is a fairly common case, and according to Thomas Christensen, the word on the street is that at many TDD companies the motto is: “We don’t fix bugs, we fix tests.” This brings us to the first risk of Chapter 2: According to Matt Stephens and Doug Rosenberg, in a typical TDD setting, half the code mass comprises unit tests. Those tests are code. Where there is code, there are bugs (see the StickyNotes for a reference). Programmers are psychologically programmed to trust their tests over their code—it takes only five minutes to write the test and an order of magnitude longer to write the code. They sometimes will make arbitrary changes to the code to make the test pass, even without understanding why the test now passes. (Sorry, it happens.) Having a high density of testing code relative to the delivered code can actually increase the number of bugs in the delivered code! To fix this: • Test at a coarser level of granularity—perhaps at some level of integration that makes business sense. That reduces the testing code mass relative to the delivered code mass. • Make sure that tests and code are independently developed. • Don’t neglect system testing. We tell our developers the story of Ada compiler development in the 1980s. It was a test-driven project driven by an acceptance suite. Most suppliers used the test suite to validate their compilers. Not Nokia. Nokia verified it against the Ada language definition. The Nokia compiler was the only Ada compiler not to exhibit the rendezvous bug that had been institutionalized in the test suite. There is currently a similar problem in the g++ test suite, in which a colleague of ours recently found more than forty bugs. The software world is starting to learn this. As of 2005, one software notable would adopt a very conventional position and say, “Having the system-level tests before you begin implementation simplifies design, reduces stress, and improves feedback (see Beck in the StickyNotes).” Over time, the system evolves and the team eventually works out the bugs. The system goes to its first customer. Things go pretty well for the first two releases, though the competition has been making good inroads into the market. But the feature velocity is slowing because it’s difficult to turn around change quickly. To check in code, the unit tests have to pass. It now takes twenty-five minutes to run the tests. Projecting into the next release, we see that the time will soon grow to an hour. In general, the ceiling is unbounded. Writing tests at the granularity of classes and member functions will create a large code mass whose maintenance will reduce your feature velocity, and for which the test times can grow without bound. In some companies we have seen these test suites grow to eight hours. The solutions: • Again, test at a coarser level of granularity. Systematically developed tests can give good coverage of business functionality and can adequately exercise code, even from a small test mass applied at fairly high-level interfaces. • If a given piece of code cannot be exercised from such a high-level interface, then remove it. It has no bearing on anything visible to any external constituencies. As a follow-on to long check-in times, the team decides to review current tests. www.StickyMinds.com They find that many of them—dozens, in fact—are now obsolete. Some of the test failures should be ignored, and some developers set some Maven flags to ignore those test results. The good news is that the team actually removes a few of the bad tests. The bad news is that they don’t create any new tests to cover the same code. The even worse news is that some of the test failures that were flagged as innocuous mask real bugs that the team will find later. After the system goes to the third customer, customers start complaining about degrading usability. The complaints seem to fall into two categories. The first complaint is straightforward: There was a new field on one of the forms, and when users tab rapidly from field to field, the habits they developed for a common case now caused the cursor to fall one field short of where it should be. The team couldn’t even understand the problem when it was faced with it. As recounted by Kent Beck (see the StickyNotes), in the spirit of Smalltalk testing, the team found changes in the interface to be only annoying details; in the spirit of agile, it is “working code” that matters. Team members also had tested this interface; they clicked on individual fields with the mouse and entered the relevant data, finding that the system presented the right answer when they pressed “Enter.” The business logic was further validated by tests at a lower level. TDD can give a false sense of confidence that the system is tested. Furthermore, it pre-ordains a certain ordering of events; doing the combinatorics is difficult and is rarely tackled by developers. The solutions: • Do system testing—or at least test at the level of meaningful business functionality—and usability testing. Treat TDD as if it has no validation or verification value— which is approximately true. • Tell system testers what they need to test. System testers have long used to their benefit well-structured use cases that don’t constrain 35 DECEMBER 2007 BETTER SOFTWARE http://www.StickyMinds.com

Table of Contents Feed for the Digital Edition of Better Software - December 2007

Better Software - December 2007 Contents Mark Your Calendar What's Happening @ StickyMinds.com Technically Speaking Code Craft Test Connection Management Chronicles Man and Machine Let Your Values be Your Guide A Story About User Stories and Test-driven Development Product Announcements The Last Word Ad Index

Better Software - December 2007

For optimal viewing of this digital publication, please enable JavaScript and then refresh the page. If you would like to try to load the digital publication without using Flash Player detection, please click here.