Better Software - July/August 2008 - (Page 42) “Within a feW hours, more than fif ty of these defects Were resolved or determined not to be defects. the cost savings Was huge (a future savings of $200,000 at $4,000 per defect) versus finding these defects the old-fashioned Way.” study attempted to determine if the tool was capable of finding the source of the rare, unexplained errors, since the consequence of these occasional errors could be large: only be asked to repair suspected defects that have been determined to be real. The time to prefilter—four hours for 200,000 lines of code—is still relatively small when compared to typical peer The ASA tool found 285 potential defects: Critical: 56 (Null pointer dereference, buffer overflows) Severe: 14 (Use of free memory) Error: 193 (Memory leaks and uninitialized variables) Warning: 21 (Inconsistent case labels) Other: 1 Within a few hours, more than fifty of these defects were resolved or determined not to be defects. The cost savings was huge (a future savings of $200,000 at $4,000 per defect) versus finding these defects the old-fashioned way. It is expected that all 285 bugs could be fixed with forty hours of developer time and that traditional methods would have taken weeks if not months to fix. Using an ASA tool requires access to the source code, external libraries, and any make and build scripts. It usually takes an hour or two to actually build the code. Prefiltering the code reports after running the build is by far the most time-consuming task. Typical prefiltering times run four hours for every 200 defects. For code with a defect rate of five per 1,000 lines, this would mean about four hours to prefilter 200,000 lines of code. Each suspected error must be categorized as either fix, not a problem, or questionable. Early experiences in handing back code directly to developers before prefiltering were unsatisfactory. Prefiltering assures that a developer will inspection times of an hour or so to inspect 300 lines of code. LLNL has analyzed more than 1.9 million lines of C, C++, and Java source code using Klocwork’s K7. To date, more than 3,000 defects or security vulnerabilities have been discovered and repaired. It is costing about $100 a defect to find defects using an ASA tool, and it is estimated that a defect would cost $4,000 if found by users after release. The bottom line is that ASA technology in the higher end tools such as Coverity Prevent and Klocwork K7 works, and the return on investment easily justifies the tool. The only down side is that some prefiltering time is necessary to eliminate the 15 to 20 percent false-positive rate, the skill set of tool users requires an understanding of the source language being analyzed, and not all of the defects found are of the show stopper variety. ASA tools support the contemporary agile approaches to software development, as these tools can be run many times in a thirty-day sprint or timebox. They also can be used in traditional environments and missioncritical applications. Contemporary ASA tools have proven to have three major advantages on our software projects: • They are capable of finding about 50 percent of all possible defect types quickly, saving time and reducing development and testing costs. • They require testers to understand computer languages. • Prefiltering is the most time-consuming task. Our future plans include insertion of static-analysis capability early in the development lifecycle during the code and debugging phases by integrating the tool into the developer’s IDE. Additional research is being conducted at LLNL on how to fix the code automatically after a defect is discovered. Because developers can see an immediate benefit of ASA tools, acceptance has been rapid. In fact, at LLNL, software quality engineering organizations are finding that developers actually are seeking out the tool once they understand what it can do for them. An ASA tool can have no better endorsement than that. {end} LLNL-JRNL-402003-DRAFT This work performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344. RefeRences: 1] Boris Beizer, Software Testing Techniques, Second Edition, Van Nostrand Reinhold, page 57, Table 2.1. 42 BETTER SOFTWARE JULY/AUGUST 2008 www.StickyMinds.com http://www.StickyMinds.com
For optimal viewing of this digital publication, please enable JavaScript and then refresh the page. If you would like to try to load the digital publication without using Flash Player detection, please click here.