Better Software - July/August 2008 - (Page 38) Table 1 Many of the defect types found by ASA tools would be difficult to find using peer group inspection techniques (the R in STAR). This is because a defect may be a combination of source-code statements that are physically far from each other in the source code—for instance, the allocation of memory and then, pages later, a return statement without releasing that memory. A human is limited in the amount of source code-detailed information that can be remembered from one page to the next. An ASA is not limited by this restriction, and besides, most of us do not relish the prospect of examining other people’s code for hours at a time. ASA tools also can find problems that elude traditional system-level testing—for example, an array-bounds overflow, where a string of twenty characters is written into a buffer of size ten. The ten memory locations that are overwritten may not manifest a failure in the program until a later execution time or may not manifest in a repeatable way. Since ASA tools are totally unaware of user requirements, the tools cannot replace the benefits of peer reviews or good functional testing. Also, ASA tools will not replace the need to use dynamic analyzers. Dynamic analyzers can find problems that occur only during interactions between the executing application code, system resources, and interfaces such as race conditions. While ASA tools are not a “silver bullet,” these tools have the capability to detect up to 50 percent of defect types and security vulnerabilities before system testing is conducted, which reduces the amount of time needed for system testing and reduces the risk of defects’ escaping to the field and being discovered by customers. Table 1, from Boris Beizer’s Software Testing Techniques [1], lists typical defect types and their percentages: ASA tools are effective on structural bugs (which are approximately 25 percent of all bugs); 66 percent of data bugs (about 14.5 percent of all bugs); 66 percent of implementation and coding bugs (about 7 percent of all bugs); and 50 percent of integration bugs (about 4 percent of all bugs), for a total of approximately 50 percent of the types of bugs found in systems. Note that ASA tools cannot find defects related to requirements, features and functionality, and testing defects. Because static analyzers use parsers, they are limited to specific source code www.StickyMinds.com languages that they can analyze. The most common languages handled by ASA tools are C, C++, and Java. Some ASA tools can handle multiple languages because they have multiple parsers. The ASA tools with the fewest false-positive rates also compile and link the source code. Reports generated by the ASA tools assume that ASA users have a basic familiarity with the source language being analyzed. ASA tools can be used for newly developed code as well as for legacy code. They can be used as a riskmitigation strategy for both open source code and vendor-supplied source code by incorporating ASA tools into an acceptance test procedure or release criteria. ASA tools also can be used on source code generated by automated code generators, such as code that is generated from graphical models. As use of ASA tools becomes widespread in software quality assurance organizations, the requirement for test engineers to become familiar with the source languages used will also increase. In the early days of software development, most of the developers (who also did the testing) were of necessity familiar with the languages used. In the last three decades, specialization has evolved into developers who write code and test engineers who test the integrated code. Black box test engineers traditionally specialized in domain-specific knowledge or business requirements and were generally discouraged from understanding what the code did. The belief was that an understanding of the code (white box) would bias the thinking of the tester. Instead, the tester should have the same perspective as the customer. ASA tools present an ideal reason for testers to upgrade their skills and learn programming languages—not so they will understand the interworking and logic of the code under test, but so they can understand the ASA tool’s defect report and explain its findings to developers in a common vocabulary. Testers can use their language skills to prefilter the ASA tool’s report to remove any false positives or defects that are superfluous in the context in which the code is used. A common example of this is a call to an error-handling routine that aborts the execution of the code. Since the error 38 BETTER SOFTWARE JULY/AUGUST 2008 http://www.StickyMinds.com
For optimal viewing of this digital publication, please enable JavaScript and then refresh the page. If you would like to try to load the digital publication without using Flash Player detection, please click here.