MSDN Magazine - November 2008 - (Page 91)

DR. JameS mccaffRey TesT Run Group Determination In Software Testing Consider the general problem of determining a best option from a list of choices, where the decisionmaking process is performed by a group of people rather than by a single person or by a purely quantitative technique. Examples of this type of activity include the residents of a state voting for the state’s governor from a list of candidates and the board of directors of a large company determining which of several smaller companies to target for acquisition. In essence, the group must rank its options in some way so that the best one can be determined. Group determination of a best alternative occurs in software development and testing, too. For example, consider a group of beta users choosing the best user interface from a set of prototypes. Or imagine the members of an open source project voting for a policy. I have observed that the practical techniques for performing group determination of a best alternative, and the potential pitfalls of the process, are not widely known in the software testing community. There are dozens of group analysis techniques. In this month’s column I describe five of the most common methods used for group determination of a best alternative in a software testing environment: the pure plurality technique, the majority runoff technique, the Borda count system, the Condorcet method, and the Schulze method. All of these analysis techniques are well-known outside of software engineering and have been extensively researched. I describe each technique with an emphasis on its application to software testing. I think you’ll find knowledge of group analysis techniques a valuable addition to your toolkit. Situations where exactly one of a set of alternatives must be selected by a group of people occur in a wide range of problem domains. Different problem domains tend to use different terminology. In sociology, for example, the study of this type of problem is usually called social choice theory and typically uses terms such as options and evaluators. In political science, analysis of these problems is often called voting theory and uses the terms candidates and voters. In mathematics, group determination of a best alternative is frequently considered a sub-branch of decision theory. I will use a mixture of terminologies in my discussion because there is no specific terminology set used when these tech- The process of performing a group evaluation and interpreting the results is somewhat trickier than you might expect. niques are applied to a software development and testing environment. The techniques I describe in this column apply to a fairly narrow range of problems and do not apply strictly to quantitative scenarios where a best choice is clearly defined. As you’ll see, when a group of people select a “best” alternative, the meaning of best is defined by the technique used. Also, the techniques I describe here do not apply to collaboration scenarios where a group of people use discussion and negotiation to arrive at a consensus decision. Let’s begin the discussion of group determination of a best alternative with a look at the simplest and most common technique—but let me warn you that this is not the best approach in many testing situations. Suppose you are developing an enterprise software application for your company’s internal use and you create four quite different user interface prototypes. Although you could ask a single person to evaluate the different prototypes, in many cases a better approach is to ask a group of people to rank the four different designs. It turns out that the process of performing a group evaluation and interpreting the results is somewhat trickier than you might expect. The simplest possible group evaluation technique is to submit the choices to the evaluators and allow them to choose, or vote for, just one alternative. The alternative that receives the most votes is declared the winner. This technique is often called the pure plurality rule. So, suppose you submit your four different prototypes (let’s call them A, B, C, and D) to 10 evaluators and ask each person to choose the best prototype. And suppose the results are: Prototype A is best according to 4 people, prototype B is ranked best by 3 people, prototype C is chosen by 2 people, and prototype D is selected best by 1 person. Therefore, you select prototype A for further development. Send your questions and comments for James to testrun@microsoft.com. Code download available at msdn.microsoft.com/magazine/cc135911. November 2008 91 Pure Plurality and Majority Runoff http://msdn.microsoft.com/magazine/cc135911

Table of Contents Feed for the Digital Edition of MSDN Magazine - November 2008

Contents MSDN Magazine - November 2008 Toolbox CLR Inside Out Data Points Cutting Edge Security Briefs Test Your Security IQ Agile SDL Access Control Utility Spotlight RIA Test Run Wicked Code Foundations Team System End Bracket

MSDN Magazine - November 2008

For optimal viewing of this digital publication, please enable JavaScript and then refresh the page. If you would like to try to load the digital publication without using Flash Player detection, please click here.