Embedded Systems Design - Europe - August/September 2007 - (Page 31)

software tools to track more subtle, insidious bugs. Error messages indicate only the symptoms of the underlying problem – they show the immediate cause of the failure. These messages tell developers the system’s state when the error is detected. But for complex misbehavior, developers need to know how the system got into the error state. That information is usually produced long before the problem is detected. In practice, error messages usually leave much to be desired. The most common problem is that the developer doesn’t put enough information into the message. Another is that crucial information relevant to the particular problem is omitted. The developer didn’t anticipate generating the message under the circumstances. Yet another difﬁculty of relying on error messages is that many types of problems don’t produce them. Developers are placed in an awkward position. There can be enormous pressure to ﬁnd and ﬁx the problem, and quickly. But the bug is invariably transient, subtle, and insidious. All of the obvious problems were found in testing prior to shipping the product. So it’s unlikely that the error messages contain the key information needed to solve problems the ﬁrst time they appear. Software-development organizations traditionally rely on two approaches to deal with transient bugs in the ﬁeld. One is to turn on “verbose logging” or “instrumentation code” at the customer site. The additional code logs a greater volume of information as the system runs. Another is to make an “instrumented” version of the software and send it to the customer. This code logs additional information that targets the speciﬁc problem being diagnosed. But both approaches suffer from a key ﬂaw – the problem must occur again! A transient problem may not occur for many months. If the product is shipping to hundreds of customers each month, a delay in ﬁnding the problem can spell disaster. It also places developers in the highly uncomfortable position of having to utter the ﬁve magic words guaranteed to infuriate a customer: “Can you reproduce the problem?” Something better is needed, a way to catch the problem the ﬁrst time it occurs. It shouldn’t require changing settings, building instrumented software, or asking a customer to reboot. FIRST-FAULT DATA CAPTURE During development, we can design a software architecture that includes its own trace data recorder (TDR) for the system. Analogous to a commercial aircraft’s “black box,” the TDR records information about the system’s dynamic behavior. After a crash, the information in the TDR helps to analyze the failure. The data can provide valuable insight into complex behavioral problems that result from sporadic changes in system inputs, unexpected interactions between subsystems, or bugs in low-level software. The TDR architecture must process “high-rate” data that would bog down a traditional logging mechanism. It’s always on, so the TDR must be fast, lightweight, and compact to keep it from affecting system performance. If using TDR functions degrade performance, developers won’t use them. At the heart of the system is a library that applications invoke to place time-tagged data into circular trace buffers in system RAM. Another TDR component, the dump agent, catalogs the buffers. In the event of a system failure, the dump agent freezes the buffers to prevent critical data from being overwritten. It then dumps the trace buffers to nonvolatile storage. After a dump occurs, a retrieval agent transports it out of the system. The dump may be analyzed using a software tool generated automatically from the system’s source code. Figure 1 illustrates a simpliﬁed architecture for a TDR system. Because disaster strikes in myriad forms, there should be several paths to trigger a TDR dump. It’s important to note that the TDR subsystem itself never triggers a dump. The dump occurs only when commanded by another software subsystem. The most valuable time to generate a dump is when software detects a catastrophic error. This dump can be 31 www.embedded.com/europe | embedded systems design europe | AUGUST – SEPTEMBER 2007 030-031-032-033-034-035_ESDE.ind31 31 5/09/07 12:31:25 http://www.embedded.com/europe

Table of Contents Feed for the Digital Edition of Embedded Systems Design - Europe - August/September 2007

Contents News Cover Feature: Annual Study Uncovers the Embedded Market DSP Serves the Convergence Needs of Small Business Embedded Test Offers Unique Value for Serial I/O The Software Detective: First-Fault Data Capture Boards May Shrink But Performance Doesn't New Products Advertising Contacts

Embedded Systems Design - Europe - August/September 2007

For optimal viewing of this digital publication, please enable JavaScript and then refresh the page. If you would like to try to load the digital publication without using Flash Player detection, please click here.