Dr. Dobb's Journal - May 2008 - (Page 45)

d05zeid_p3db 3/13/08 9:35 AM Page 45 increasing correlation values, but were not necessarily signs of plagiarism. Had these results been known upfront, some of them could have been eliminated before CodeMatch was run. However, given the number of files and the number of source-code elements, it was impractical to find these elements before performing the correlation. Also, the correlation itself pointed out many of these superfluous elements. My experience using filtering with CodeMatch can be generalized to any kind of information retrieval process. To understand how filtering can be used, it is important to first understand the different kinds of information retrieval processes and information display methods. Discover a New World in Programming Information Retrieval Since the best-known information retrieval process is probably web searching, I use it as an example. Information retrieval can be classified into two types—exact match and best match. The exact match type of information retrieval is represented by the Boolean retrieval method. In these cases, Boolean equations of keywords are entered by users and all objects in the information domain that meet the criteria are retrieved. Most search engines use exact matching; the information domain is the Web. Even the more sophisticated search engines that let users input natural-language queries are typically parsing the language to retrieve the keywords and Boolean equations. Best-match information retrieval uses vector space and probabilistic retrieval methods that essentially try to understand what information a user wants, sometimes based on past searches or other stored user parameters, and then present the information to the user that seems closest to what the user wants. An example of this would be the book suggestions that Amazon.com presents to customers, based on the customer’s search criteria and past searches. Figure 1 is a graphical representation of information retrieval, where D is the information domain, Q is the user's query, and O is the object retrieved by the query. DU is the subset of the domain that meets the user’s information need based on the retrieval process. Each arrow from an object to the query represents the relationship Ri between the query and object. For all retrieval methods, DU is the set of all objects, such that Ri>0: DU={Oi : Ri > 0 for all i} CodeMatch Post-Process Filtering To make examining the correlation results more useful and let users focus on the kinds of correlation that are most important, I added the ability to filter the results. After CodeMatch produces a database of results, this filtering can be performed on the database: • Statement filtering. A list of statements is created by users. Any correlation due to a statement on this list is eliminated. • Comment filtering. A list of comments is created by users. Any correlation due to a comment on this list is eliminated. • Identifier filtering. A list of identifiers is created by users. Any correlation due to an identifier on this list is eliminated. • General file filtering. A list of file names is created by users. Any correlation between any file whose name appears on the list and any other file is removed from the results database. • Specific file filtering. A list of file names with specific paths is created by users. Any correlation between a specific file on the list and any other file is removed from the results database. • Folder filtering. A list of folders is created by the user. Any correlation between a file in a folder on the list and any other file is removed from the results database. • Threshold filtering. Users can change threshold parameters, reducing the number of correlated file pairs that are displayed. Users can set minimum and maximum correlation scores to display and can set a maximum number of correlated files to display. After the filtering is performed on the database, the correlation scores between file pairs are adjusted accordingly. I found that for large file sets, this filtering reduced the manual process of reviewing the results in order to find plagiarized source-code files from days to hours or even minutes. Looking for a career with an innovative company where you can use your programming skills to make a difference in the world? Join ESRI and design and develop the next generation of our world-leading geographic information system (GIS) mapping software. We are seeking software developers with solid core programming skills and a passion for inventing new technology. We have opportunities to work on everything from database and Web development to graphics, 2D–3D rendering, and core server technology. Our developers work with the latest technologies including Java, .NET, and Flash/Flex. At ESRI you’ll have exceptional benefits, opportunities for growth, and the satisfaction of developing software that is making a difference. Join us and be a part of changing the world. Learn more about ESRI and apply online at www.esri.com/ddj. However, the relationship can be further refined depending on whether the retrieval Copyright © 2007 ESRI. All rights reserved. ESRI, the ESRI globe logo, and www.esri.com, are trademarks, registered trademarks, or service marks of ESRI in the United States, the European Community, or certain other jurisdictions. Other companies and products mentioned herein may be trademarks or registered trademarks of their respective trademark owners. ESRI is an Equal Opportunity Employer. May 2008 l www.ddj.com l Dr. Dobb’s Journal 45 http://Amazon.com http://www.esri.com/ddj http://www.esri.com/ddj http://www.ddj.com

Table of Contents Feed for the Digital Edition of Dr. Dobb's Journal - May 2008

Dr. Dobb's Journal - May 2008 Contents Friday Night Fish Fry Alia Vox Developer Diaries Software Development Goes to the Movies Cat: A Functional Stack-Based Little Language Mojax: Mobile Ajax Framework Kernel-Mode Databases Getting Better Search Results Effective Concurrency The Agile Edge

Dr. Dobb's Journal - May 2008

For optimal viewing of this digital publication, please enable JavaScript and then refresh the page. If you would like to try to load the digital publication without using Flash Player detection, please click here.