Better Software - December 2007 - (Page 14)

Test Connection What Counts? by Michael Bolton A while ago, I was asked to test a network packet sniffer. The system displayed a pie chart indicating the relative number of packets associated with each IP address that had been accessed during the past five minutes. I generated some traffic and observed that for five of the sixteen IP addresses that I had tried, the display was incorrect—the pie chart overwrote the legend identifying the IP address. Was that one test resulting in one failure or sixteen tests with five failures? I noted that the relative proportions of each slice of the pie seemed to be correct. Then I added one more IP address and noticed the colors of the pie sections changed (they were supposed to stay the same). Now, was that one test or two? If one, then the single test failed. If two, did the first test pass and the second test fail? I noticed that the pie chart appeared to be a circle. When I resized the window by shrinking the y-axis, the circle got smaller. When I restored the window to its previous size, the circle returned to its original size. When I resized the window by shrinking the x-axis, the circle got smaller. When I resized it by expanding the x-axis, the circle again returned to its original size. Was that one test (“resize the window”), two tests (x, y), or four tests? While I was investigating the colorchanging bug, I checked the specification and found that the pie chart window was supposed to be a fixed size and not resizable at all. This new information made me realize that my earlier sizing tests had uncovered a bug, which changed my perspective on the tests that I had already performed. Was that one failing test (one bug) or four failing tests (four bugs)? To make sure I had really observed a problem, I tried resizing the window again. I wondered if this new test should be counted along with the original test (tests?). A couple of years ago, I wrote an automated test for an application that checked transaction types. A four-digit field became the index into a table of transactions. About 3,000 slots in this table were populated and valid; the other slots were not. An attempt to use a populated entry was considered valid while an attempt to use an unpopulated entry was invalid. I wrote a quick bit of script to test all 10,000 possible values. One of the populated entries was inappropriately rejected. Did I do 10,000 tests (“test each value”), two tests (“test all valid values”; “test all invalid values”), or did I do one test (“test all values”)? Note that, in order to run this test (tests?), I had to populate eight other fields with plausible data for each transaction, and creating the values in some of those fields was challenging. Not only that, but they were to some degree randomized, so I was testing certain combinations. Should I have multiplied 10,000 by eight for a total of 80,000 tests? I also observed the time it took to complete transactions (turnaround time ranged from just under half a second to 1.5 seconds). Was that observation a test? Should I have added another 10,000 test cases to my count? I’m raising these questions not because I’m some wise guy who enjoys teasing people with obscure corner cases of philosophy and logic. These aren't corner cases. Deciding what to count and how to count it are the kinds of decisions that we make every day. But more importantly, I’m raising these questions because counting tests (and requirements, bugs, and other measures derived from these counts) is an endemic means of deception in the testing business. Some well-known testing experts promote this form of deception; testers then practice it, and project communities have learned to ask for it. The problem here is reification, the act of giving construct attributes to things that are merely concepts. Test cases, bugs, and requirements are expressions of ideas. If we were evaluating the work of a film critic, we would sound foolish if we were to ask how many observations she made, how many problems she had found in the film, or how many paragraphs she had written. I observe that in the testing business, we are infected with counting 14 BETTER SOFTWARE DECEMBER 2007 www.StickyMinds.com ISTOCKPHOTO http://www.StickyMinds.com

Table of Contents Feed for the Digital Edition of Better Software - December 2007

Better Software - December 2007 Contents Mark Your Calendar What's Happening @ StickyMinds.com Technically Speaking Code Craft Test Connection Management Chronicles Man and Machine Let Your Values be Your Guide A Story About User Stories and Test-driven Development Product Announcements The Last Word Ad Index

Better Software - December 2007

For optimal viewing of this digital publication, please enable JavaScript and then refresh the page. If you would like to try to load the digital publication without using Flash Player detection, please click here.