Better Software - January 2009 - (Page 11) Technically Speaking The TSA and Software Testing by Lee Copeland “TSA screener testing labeled ‘a waste’” is the USATODAY.com headline [1]. The article states, “A government program to find gaps in airport screening is ‘a waste of money’ because it doesn’t follow up on why screeners failed to spot guns, knives, and bombs on undercover agents,” the head of the House Homeland Security Committee says. “You have a system that’s supposed to strengthen airport security, but you don’t use the results of the tests to do exactly what you’re doing the tests for,” said Bennie Thompson, D-Miss. “It’s obviously a waste of money.” If this story seems vaguely familiar, just substitute a few nouns. Replace “TSA” with “your organization”; replace “bombs” with “software defects,” and now it becomes clear. Most organizations fail to learn very much from the information that testing provides. centage of the number of detest cases is substanfects detected by all testing tially easier. A key If a test case efforts divided by the total metric is test coverage number of defects found (the percentage of the never finds a (both by testing and by custest object that has tomers in the field). DDPs been “covered” by the in the range of 0 percent test suite). Coverage is defect, does that mean it to 50 percent indicate that generally defined at the our testing process is very requirements level and is a poor test? ineffective. DDPs above 90 at the code level. While percent indicate our testing low coverage numbers (0 percent to 50 percent) frighten me, is effective (only one out of ten defects high coverage numbers (80 percent to is escaping into the field). However, for 100 percent), while impressive, don’t some products and industries, DDPs of 99 percent or more are required. guarantee defect-free code. Once this simple form of DDP is For example, with only one test case, we can achieve 100 percent code cov- measured, used, and institutionalized, consider breaking “testing” into levels erage on this simple program: such as unit, integration, system, and Int blech ( int p, int q) { acceptance and measuring DDP at each return p/q; level. If, for example, the DDP of unit } But it contains a fault—it will fail testing was significantly lower than the other DDPs, this could indicate that the when q is 0. unit tests were not well designed or implemented. On the other hand, it could Measuring the Quality of indicate that the units themselves are the Development Process The great quality guru Edwards De- well defined and well implemented, but ming believed that approximately 80 the interfaces are poorly defined and so percent of all product defects come from defects are found later in testing. One final word on measuring—the the process being used, while only 20 percent come from individual worker’s metrics, the numbers themselves, rarely mistakes. The results of testing can tell point directly at the problem. Often, us much about the quality of the devel- they are more like a red flag waving to opment process. For example, if a cer- indicate that there is a problem sometain type of defect is created again and where. Before jumping to conclusions again, those defects are not coming from about the problem and its potential individual mistakes; they are coming solution, remember Jerry Weinberg’s from the development process itself. “Rule of Three”: If you can’t think Deming explains who is responsible for of at least three different interpretaimproving the process. He said the staff tions of a metric, you haven’t thought works within the process; it’s manage- enough about what it might mean. And ment’s responsibility to work on the pro- that means you haven’t thought enough about the solution. {end} cess to improve it. Measuring the Quality of the Product Almost everyone uses the results of testing as an indicator of product quality—many defects usually means poor quality; few defects, well, that’s more complicated. It could mean that the product is of good quality. It could also mean that our testing is not effective in finding the defects. But, there is much more to be learned from the results of testing. Measuring the Quality of the Tests Measuring the quality of an individual test case is difficult. If a test case never finds a defect, does that mean it is a poor test? While some testing experts say yes, the fact that the test case passed (assuming it is well designed, implemented, and executed) provides valuable information. It tells us there are no defects in the area it has tested. And knowing that lets us focus our limited resources elsewhere. Measuring the quality of a suite of Measuring the Quality of the Testing Process One of the most effective metrics for measuring the quality of the testing is defect detection percentage (DDP). In its simplest form, DDP is defined as the perwww.StickyMinds.com RefeRenceS: [1] Frank, Thomas. “TSA Screening Labeled a Waste.” USA Today. November 14, 2008. www.usatoday.com/travel/flights/2008-08-13tsatests_N.htm JANUARY/FEBRUARY 2009 BETTER SOFTWARE 11 http://www.USATODAY.com http://www.usatoday.com/travel/flights/2008-08-13-tsatests_N.htm http://www.StickyMinds.com
For optimal viewing of this digital publication, please enable JavaScript and then refresh the page. If you would like to try to load the digital publication without using Flash Player detection, please click here.