Better Software - March 2008 - (Page 18)

Test Connection How Much Is Enough? by Michael Bolton experienced might not be able to articulate the elements of the stopping decision, but the skilled practitioners can. Exploratory testers design and execute tests in the moment, starting with an open mission and investigating new ideas as they arise. When do we stop? How can we be sure that we were done? And how can we justify our decisions to our clients? The first step is to recognize that we can’t be sure that we’re done. Any approach to answering the stopping question is necessarily heuristic, based on some fallible, context-dependent method for solving a problem or making a decision. Heuristics serve the purpose of learning and discovery quickly and inexpensively, but not conclusively— rather like indicator lights on your car’s dashboard. When any lamp on your dashboard glows red, you should consider stopping until they’re all green again—but maybe it’s more important to ignore the lamp and keep going. Consider these heuristics: Time’s Up!: If we’ve specified an explicit timebox for a test, we might choose to stop when we’ve exhausted the time that we allocated to the task. Might our timebox have been inappropriately arbitrary? Might we want to allocate more time for further investigation? Piñata: When the candy starts cascading from a piñata, we typically stop whacking it. If we see something reasonably spectacular when we’re testing—a crash or hang, screen or data corruption, behavior that we can’t understand or explain—we might believe that we’ve www.StickyMinds.com “I know how to spell banana,” the little girl said. “I just don’t know when to stop.” An entirely scripted action is one in which the person performing it makes no decisions of her own; all of the ideas came from some other person, some time in the past. An entirely exploratory task comes from inside the person performing it, in the moment that the task is being performed. No test performed for a client is entirely exploratory: The exploratory tester has a mission that ultimately derives from some goal or need previously expressed by the client. No good test—except one performed by an automaton—is entirely scripted, either. Only the worst imaginable human tester will fail to break out of the script, pause, and take note when he recognizes a problem of some kind. Even in a heavily scripted process, testers are usually encouraged to investigate the bugs that they find—and that requires exploratory behavior. Since the bug itself was unpredicted, the decision to stop is unpredictable, too. The nice thing about an entirely scripted task is that you know when to stop. The last note of the musical score, the last line of the play, and the last empty space on the paint-by-number canvas all mark the end of the work. When you’re using a highly scripted approach to test, you stop when you’ve noticed a problem, have more questions, or have an interesting idea. When you’re exploring, knowing when to stop is less obvious. In a jam session, musicians signal each other with eye contact and musical cues so that they can fade out or stop together. Improv actors build unscripted scenes, play them out, and then freeze in position or break character when they discover a plausible conclusion. To someone unskilled in these arts, the factors that go into the decision to stop tend to be unclear or invisible. Even among performers, the less 18 BETTER SOFTWARE MARCH 2008 found a problem that’s dramatic enough to justify stopping. Might there be an interesting piece of candy still stuck in the piñata? Dead Horse: When the application is down for the count, there’s no point in continuing to test. Anything we see after this is likely to be a side effect of things that we’ve already discovered, or our system is corrupted and the integrity of the test is compromised, as the developers will quickly point out when we try to report. Moreover, the problems that we’ve already discovered will lead to bug fixes, and those changes will require more testing later. Might we see an even more dramatic or damaging problem if we proceed a little further? Loss of Charter: Ultimately, we work in the service of a client. If we have any reason to believe that our work no longer has the support of the client, then we’ve lost the primary motivation for our test. If the client asks us specifically to stop, we’re mandated to do so. Have ISTOCKPHOTO http://www.StickyMinds.com

Table of Contents Feed for the Digital Edition of Better Software - March 2008

Better Software - March 2008 Contents Mark Your Calendar Contributors eLightenment Technically Speaking Code Craft Test Connection Management Chronicles Cover Story: Breaking Ground On SOA Software Development Worst Practices Mind the Gap Product Announcements 10 Things You Might Not Know About... The Last Word Ad Index

Better Software - March 2008

For optimal viewing of this digital publication, please enable JavaScript and then refresh the page. If you would like to try to load the digital publication without using Flash Player detection, please click here.