Better Software - December 2007 - (Page 15)

Test Connection “I observe that in the testing business, we are infected with counting disease—we are constantly counting test cases, requirements, lines of code, and bugs. ” disease—we are constantly counting test cases, requirements, lines of code, and bugs. Yet surely one bug can be dramatically different from the next by all kinds of dimensions—the impact on a given user, the proportion of the user base it affects, the extent to which it blocks our ability to see other bugs, the time it takes to pinpoint it, and the effort required to fix it. In a similar way, the value of two tests can differ dramatically. If this test compares an output value with some reference, and that test measures the response time for a Web transaction that involves dozens of interactions with a browser, networks, and databases, should we consider them equivalent and count them as the same? Counting tests is like counting vehicles or cargo—where a vehicle might be a bus, freight train, tricycle, or space shuttle, and cargo might be passengers, wheat, televisions, or nuclear waste. The problem is compounded by dividing these counts by other counts to create ratios, as though cargo per vehicle were a meaningful ratio unless we knew much more about the cargo and about the vehicle—to the point where the ratio might be the least interesting thing about it. Another fallacy with counting tests is the assumption that test results are binary—pass or fail. But that is not the question that we really want to ask as testers. That is important, but if we pay too much attention to that question, or consider that question only, we’re likely to miss problems. Instead, ask a more fundamental question: Is there a problem here? When I’m performing tests—configuring, operating, observing, and evaluating an application—I'm making dozens of observations every second. Some of them are important, some trivial (“That’s a button; now it’s highlighted; there’s a tooltip; that word is spelled correctly…”) Sometimes I’m not even terribly conscious of making them until, all of a sudden, something catches my attention. (“Hey, the tooltip didn’t go away.”) In the course of preparing for a test of some condition, I often observe a failure of some other condition for which I hadn’t intended to test. (“The screen isn’t redrawing properly.”) Often, my observation appears to be inconsistent with some expectation that I didn’t even realize I had until the inconsistency appeared. When I take a car for a fifteen-minute test drive, I make evaluations of all kinds of stuff, paying real attention to what impresses me or bugs me. Those evaluations are the result of hundreds of conscious and subconscious tests. How do I enumerate those tests— and would it be meaningful for me to do so? At the very least, we should not deceive ourselves by counting test cases. If your management asks you to do so, you may choose to provide them with the misleading information for which they are asking. It is up to you, but I guarantee that your management does not understand what “742 test cases” actually means. I don’t know what it means, and I’m an expert in this stuff. In a relatively obscure Monty Python sketch, an interviewer asks a seasoned but clearly incompetent actor about the hardest role in the theatre. The veteran responds that the answer must be Hamlet—Hamlet has 8,262 words. Othello is hard, too—but it has 941 fewer words than Hamlet. The interviewer learns quickly: “How many words did you have to say as King Lear at the Aldwych in ’52?”—but the actor cautions him, “Well, now, I don’t want to give you the impression that it’s simply the number of words. Getting them in the right order is just as important.” Testers do the same thing when they count test cases and requirements, or when they compare the number of automated test cases with the number of “manual” ones—with the implication that tests are performed by machines or hands, rather than by minds. Worse, just as the doddering actor did, testers train their managers, project communities— even themselves—to accept those deceptive and seductive numbers. {end} Michael Bolton lives in Toronto and teaches heuristics and exploratory testing in Canada, the United States, and other countries. He is co-author, with James Bach, of Rapid Software Testing and a regular contributor to Better Software magazine. Contact Michael at mb@developsense.com. Do numbers really speak for themselves, or do they need a credible story to give them life and meaning? M Follow the link on the StickyMinds.com homepage to join the conversation. Log on to www.StickyMinds.com to read and comment on all of the Code Craft, Test Connection, and Management Chronicles columns. www.StickyMinds.com DECEMBER 2007 BETTER SOFTWARE 15 http://StickyMinds.com http://www.StickyMinds.com http://www.StickyMinds.com

Table of Contents Feed for the Digital Edition of Better Software - December 2007

Better Software - December 2007 Contents Mark Your Calendar What's Happening @ StickyMinds.com Technically Speaking Code Craft Test Connection Management Chronicles Man and Machine Let Your Values be Your Guide A Story About User Stories and Test-driven Development Product Announcements The Last Word Ad Index

Better Software - December 2007

For optimal viewing of this digital publication, please enable JavaScript and then refresh the page. If you would like to try to load the digital publication without using Flash Player detection, please click here.