Enterprise Search Sourcebook 2008

crawlers have to know how to do incremental updates from the source repository by either keeping its own database of documents that it has seen before, or by keeping some kind of high-water mark. Most enterprise search vendors supply crawlers that use a modular pipeline approach. There is a component that interacts directly with the source repository API. In the case of a web crawler, this component makes HTTP requests and retrieves webpages. It then parses the webpage for links contained within the webpage and adds them to the list of pages to download and index. File system crawlers are much simpler—usually recursively walking a directory structure and not bothering to parse the files for links. Document management system crawlers, on the other hand, use a variety of techniques, the most common of which is to use the native API of the document management system and retrieve documents according to rules set forth in the crawler configuration specific to that repository. Another technique is to create a simple web front-end page that contains a site map and templates, which render the document management repository as a set of simple webpages. These webpages can then be crawled by the HTTP module. Finally, databases are the most flexible of all data sources. Since good normalized database design is an impedance mismatch with good full-text index design, your well-normalized database must be denormalized into a flat table of “documents.” There are three general techniques for database indexing: 1. Build SQL or stored procedures on the database side, which denormalize the tables into a single “documents” table. 2. Use database connectivity from the source language (e.g., JDBC for Java) to retrieve the individual component tables and do the “join” in your program code. 3. Build HTML or XML templates hosted on a website, which renders the database information as “pages” that can be crawled by the HTTP crawler. In a way, it is kind of nice that Solr does not impose its own method of crawling data. In my experience, crawlers are often the weakest part of a commercial full-text search system. They are the redheaded stepchildren of companies that build fancy commercial systems. It is an unglamorous job to write these things, and all the cool kids want to work on the kernel. Although there is no built-in crawler for Solr, there is a loose integration between the Nutch crawler and Solr. Currently, this integration is implemented as a patch for the Nutch code base. In order to use it, you must build Nutch from scratch and apply patches. In the open source community this is seen as no big deal. It is not a big deal for open source Java geeks, so if you are one of those, right on! To me, it’s kind of a pain in the neck, and it’s one side of the double-edged sword that is open source software. Bells, Whistles, and Spangles Solr is a big step toward true enterprise search, but it’s not quite there yet. If you are expecting a complete out-ofbox experience for indexing and searching your enterprise, then look elsewhere. While Lucene, Nutch, and Solr are quite sophisticated and are not much more difficult to install than equivalent closed source software, you sometimes may have to dive into the arcane world of tools like Subversion, SOLR is for enterprises that Have access to Java programmers either in-house or as consultants. Have an understanding of their search domain and are willing to study up a bit on some pretty advanced concepts. Are willing to write some code or repurpose some existing code to get documents or data into the index. Are willing to write code for the front-end of the search engine (this is required for all search engines, not just SOLR). Either have no search access control requirements, or are confident in implementing them at the search-engine level. Want faceted search. Are willing to accept the terms of the Apache License 2.0. Need exceptional control over scoring and ranking algorithms. Want to add their own scoring and ranking algorithms. Have an enterprise mandate to use open source software when available. Do not have organizational biases or restrictions against using open source software. SOLR is not for enterprises that Don’t have access to Java programmers. Don’t have a strong business case for using a sophisticated enterprise search platform. Don’t want to write code for crawlers or connectors. Have complex access control requirements that cannot be encoded easily into search fields for filter queries. Have organizational biases or restrictions against using open source software. Require unsophisticated search or very few documents that can be indexed by one of the many free or inexpensive commercial search tools available. WWW.ENTERPRISESEARCHCENTER.COM 33 http://WWW.ENTERPRISESEARCHCENTER.COM

Table of Contents Feed for the Digital Edition of Enterprise Search Sourcebook 2008

Enterprise Search Sourcebook 2008 Contents Editor’s Note Publisher’s Note Findings and Figures Why Enterprise Search Will Never Be Google-y Searching for Search Usability Your Users Are Talking to You What’s Your Search Story? Search Is Dead—Now What? Delivering on the Promise of Enterprise Search Taming Multiple Search Engines in Your Organization Enterprise Search: Trends for 2008 Enterprise Search Seen From the Inside Open Source Search: Elixir or Poison? Avoiding the Big Mistakes in Search Semantic Search Takes Root in the Enterprise E-Discovery Essentials: The Rules You Need to Know SharePoint Search: An Enterprise Contender? Integrating Security Into Your Enterprise Search Infrastructure Engineering a Better Search Infrastructure Letting End Users Ask the Questions, Stat! The Power of Knowledge Legal Research Using Enterprise Search: A Developer’s View From Treading Water to Full Steam Ahead Pulling Out All the Stops With Midas A Natural Search Solution An Incremental Approach to Improving Enterprise Search The Enterprise Search Sourcebook Showcase Directory Index to Advertisers and Companies Mentioned

For optimal viewing of this digital publication, please enable JavaScript and then refresh the page. If you would like to try to load the digital publication without using Flash Player detection, please click here.