Enterprise Search Sourcebook 2008

Ant, Maven, and Eclipse to make your own build of the software. This can be daunting even for grizzled veterans since there are a lot of different ways to do it, and all the open source guys (yes, they are mostly guys) have their own idea of what constitutes a good build environment. One thing you can do is index and search a huge number of documents into Solr. The theoretical limit is the size of a big integer, about 2 billion documents. That’s a lot, even for commercial engines. Of course, if you are really thinking of indexing that many documents, you should consider Nutch rather than Solr. Nutch is more suited to web-scale applications than enterprise-scale applications. While I’ve not personally tried to index and search that many documents, the confident chatter on the message boards is that 30 million documents is a walk in the park for Solr. You probably don’t have 30 million documents to index, but it is comforting to know that you could. In the Schema Things As I mentioned before, Lucene is a low-level library. All information in Lucene is stored as text even if the values could be interpreted as integers, floating points, dates, or even custom data types. It lacks any kind of schema. This means you can’t do arithmetic comparisons, numeric range searches, or date-parameter searches—which can seriously limit the effectiveness of search in the enterprise. Solr enhances Lucene by adding sophisticated schema support. Not only does this schema support common data types, but you can also define your own custom data types, as well as dynamic fields whose names match a specific pattern. For example, you can designate all fields that end with *_s to be treated as string fields. This gives you some flexibility if you don’t know the types of all of the fields that are being added to the index beforehand, yet you still want to represent different fields in different ways. Configuration in Solr is handled by a schema configuration XML file, which allows you to specify advanced Lucene analyzers for each field type. You can also specify things like stemming, lemmatization, synonyms, stop-word lists, and sounds-like filters in the configuration file. Solr supports “keyword in context” results highlighting, advanced query caching, and index replication and integration with the Luke index analyzer toolkit. It has a nifty feature called “copy fields” which allows you to treat the same incoming data in different ways for different purposes. You will find all of these features in the best commercial software packages in varying mixes. It is surprising to see such a complete set of features this early in the release cycle of an open source software project. Licensed to Drive Technology isn’t the only difference between open source and vendor-based solutions. Remember—open source is not public domain. One of the practical problems for any organization considering the use of open source FEATURE API LUCENE Java-based traditional API. NUTCH Java-based traditional API. SOLR REST (HTTP/XML). Wrappers for Java, Ruby. CRAWLER None—roll your own. Web and file system crawler included. Accepts XML documents in a specific schema. Also accepts comma-separated lists of documents. FILTERS None—roll your own. Built-in support for HTML, text, Microsoft Office, OpenOffice, and PDF. Roll your own filters with the plug-in architecture. None—You convert from the source format to XML or CSV. SCHEMA None—all data is stored as text. You are responsible for converting to other data types. Fixed. Fully configurable schema. You can specify field types and storage methods declaratively in a configuration file. PLUG-IN MODULES None—roll your own by subclassing and implementing interfaces. Trés POJO. Plug-in system is based on the one used in Eclipse 2.x. Plug-ins are central to how Nutch works. All of the parsing, indexing, and searching that Nutch does is actually accomplished by various plug-ins. Plug-in code can be loaded into Solr by putting Jars containing your classes in a “lib” directory in your Solr Home directory prior to starting your servlet container. 34 ENTERPRISE SEARCH SOURCEBOOK 2008

Table of Contents Feed for the Digital Edition of Enterprise Search Sourcebook 2008

Enterprise Search Sourcebook 2008 Contents Editor’s Note Publisher’s Note Findings and Figures Why Enterprise Search Will Never Be Google-y Searching for Search Usability Your Users Are Talking to You What’s Your Search Story? Search Is Dead—Now What? Delivering on the Promise of Enterprise Search Taming Multiple Search Engines in Your Organization Enterprise Search: Trends for 2008 Enterprise Search Seen From the Inside Open Source Search: Elixir or Poison? Avoiding the Big Mistakes in Search Semantic Search Takes Root in the Enterprise E-Discovery Essentials: The Rules You Need to Know SharePoint Search: An Enterprise Contender? Integrating Security Into Your Enterprise Search Infrastructure Engineering a Better Search Infrastructure Letting End Users Ask the Questions, Stat! The Power of Knowledge Legal Research Using Enterprise Search: A Developer’s View From Treading Water to Full Steam Ahead Pulling Out All the Stops With Midas A Natural Search Solution An Incremental Approach to Improving Enterprise Search The Enterprise Search Sourcebook Showcase Directory Index to Advertisers and Companies Mentioned

For optimal viewing of this digital publication, please enable JavaScript and then refresh the page. If you would like to try to load the digital publication without using Flash Player detection, please click here.