Dr. Dobb's Journal - August 2008 - (Page 51)

a potential architectural solution that addresses that scope. The greater the detail of your specification efforts, the greater your ability to accurately estimate the effort required. Second, you glean critical sizing measures from these specifications. Third, you take measures from historical project data, ideally from your own organization. Fourth, you put all of these measures into a “magic formula,” which gives you your estimate. The measures and formula vary between estimation techniques, but the high-level process remains the same. As a result, these formal cost estimation strategies all suffer from a common set of challenges: • We don’t seem to be good at estimation in practice. The Standish Group’s “Chaos Report” (www.standishgroup .com) found that only 31 percent of IT projects are reasonably on time and on budget and that on average projects cost 189 percent of what was originally estimated. These results belie the beliefs that fixed-price projects put a maximum on the financial outlay and that it’s possible to reasonably compare projects based on their initial estimates. This makes it difficult to identify the potential ROI of an IT project, to compare the responses to an RFP, or to perform reasonably accurate annual budgeting. • We don’t seem to be very good at estimation in theory either. In 1987, Chris Kemerer reported in “An Empirical Validation of Software Cost Estimation Models” (Communications of the ACM, May 1987) that formal estimation techniques had error rates of between 85–772 percent when applied in an uncalibrated manner. So, if you don’t have historic data specific to your organization, you’re likely in serious trouble. However, Kemerer found that with significant calibration for your environment, often increasing your overall planning costs past the point of diminishing returns, estimates became 88 percent accurate. The bad news is that this study was done during the 1980s, a period where we had significant experience with the dominant technologies of the time (COBOL, C,…). This paper is just an exemplar; there are dozens of others that show that formal estimation strategies do not work as well in practice as some of their protagonists may claim. • There isn’t a very good connection between the theories being espoused and actual practice. Magne Jxrgensen and Martin Shepperd reported in “A Systematic Review of Software Development Cost Estimation Studies” (IEEE Transactions on Software Engineering, January 2007) that there still needs to be significant research into the practical application of software estimation techniques, particularly in real-world settings as opposed to arbitrary data sets. Although they included several hundred references to estimation papers, they couldn’t find a single study that examined the application of formal software cost-estimation techniques in a real-world setting. There were many that used actual historical data, but no study explored how the estimation strategies were applied in practice. In short, there appears to be a lot of blind faith in the veracity of up-front estimation techniques. • Everyone isn’t created equal. As early as 1968, Sackman, Erikson, and Grant reported a 20-to-1 productivity ratio between the best and worst programmers (“Exploratory Experimental Studies Comparing Online and Offline Programming Performance,” Communications of the ACM, January 1968). Since then, flaws in their methodology have been pointed out, and arguably a ratio of 10-to-1 is more realistic. More recently, Barry Boehm reported a 10-to-1 ratio in Software Cost Estimation with Cocomo II (Addison-Wesley 2000). The point is that unless you know exactly who will be doing the work, or at least have a way to guarantee the category of people who will work on the project, then your historic data proves questionable. • Customers know we can’t estimate up front effectively. Dr. Dobb’s 2007 Project Success Survey (see “Defining Success,” www.ddj.com/architect/202800777) found that 87 percent of business stakeholders are clearly more interested in achieving good ROI than coming in under budget. They might be asking us for fixed-price projects, but that’s not really what they want—begging the question why we’re even on this fixedprice treadmill at all. • Historical data isn’t accurate. Formal estimation techniques rely on historical data, and even when that data exists it is rarely accurate. Having a decade or more of historical data is of little value when the technologies that you’re working with change every couple of years, August 2008 l www.ddj.com l Dr. Dobb’s Journal 51 http://www.ddj.com/architect/202800777 http://www.standishgroup.com http://www.standishgroup.com http://www.rallydev.com http://www.rallydev.com http://www.ddj.com

Table of Contents Feed for the Digital Edition of Dr. Dobb's Journal - August 2008

Dr. Dobb's Journal - August 2008 Contents Friday Night Fish Fry Alia Vox Developer Diaries Developer’s Notebook A Conversation with Christos Papadimitriou OpenGL and Mobile Devices: Round 2 Ellipse Specification Using Vectors Embed Custom GUIs in WPF Building RIAs on J2EE Foundations Disentangling Concepts in Object-Oriented Systems The Agile Edge Effective Concurrency Swaine’s Flames

Dr. Dobb's Journal - August 2008

For optimal viewing of this digital publication, please enable JavaScript and then refresh the page. If you would like to try to load the digital publication without using Flash Player detection, please click here.