Enterprise Search Sourcebook 2008

or email client then you can use Lucene as is. Enterprise search, however, requires a lot more software. Ultimately, Lucene is just a library. All of you librarians are out there saying “Hey, wait a gosh-darn minute. Just a library? Libraries are the backbone of modern civilization.” I also hear you mutter “what a poltroon” in that passiveaggressive librarian tone of yours. Don’t tell me I’m hearing things. And yes, I do know what “poltroon” means. Solr is a subproject of Lucene. Apparently, it follows the Web 2.0 hipstr trend of eliminating those redundant schwa ’s from our vocabulary. if ur <30 u know what i mean lol. jk. The Solr developers define it better than I do, so here goes: “Solr is an open source enterprise search server based on the Lucene Java search library, with XML/HTTP and JSON APIs, hit highlighting, faceted search, caching, replication, and a web administration interface. It runs in a Java servlet container such as Tomcat.” In other words, Solr adds infrastructure on top of Lucene that makes it much easier to implement in an enterprise context. This is pretty much the way it works in the commercial world, too. For example, Verity had a K2 layer on top of the VDK libraries. Autonomy has an IDOL layer on top of the DRE core engine. Some other commercial engines actually use Lucene under the covers. Solr uses the buzzword-compliant REST (REpresentational State Transfer) as its communications protocol. It is implemented as a J2EE application (war file), which takes input from a standard HTTP query string and produces results in a standard XML document. This is a simple protocol that is getting good traction in the Web 2.0 community. For example, to run a query against Solr, you set up a URL with parameters and send it to the Solr server. The Solr server in turn returns an XML document, which you can then parse with any programming language known to man. If It Was Easy, Everybody Would Do It As with any enterprise product, the search tool is only part of the equation. An enterprise search tool is not very helpful unless you can extract data from the source repositories and get it into the engine in the first place. A director at an enormous search company once told me, “It’s all about connectors.” Indexing and searching are relatively easy. The hard part is getting access to every organization’s odd mix of document repositories, crawling those repositories, preserving access control information, and filtering the documents into a standard set of metadata. No matter how vanilla you think your back-office structure is, it is unique. No two organizations have precisely the same mix of content sources. It amazes me how often a client asks me something like, “How do your other customers index their custom Lotus Notes/Oracle hybrid web application on OS/2 with IBM BookMaster source files?” I generally respond with a blank stare. “Don’t you see?” the well-intentioned client stammers on, “We can just do what they do!” 32 ENTERPRISE SEARCH SOURCEBOOK 2008 I nod with the conviction of someone who just signed a statement-of-work. “See how we’ve made your life easier? We must be your best customer ever.” “Yes, yes you are. Thank you for that.” I say this with no hint of sarcasm, because I’m a professional that way. Not incidentally, this is where the enterprise search vendors make the big bucks. Mapping all of those different content repository sources into a single unified full-text index is hard work. Don’t even get me started on security. Making the Connection So, if you want to use Solr, you need to find a way to get e High-level overview of the Solr request/response architecture. Solr uses REST as its communications protocol. Requests are made with HTTP and responses are returned as XML. , your data into it. Adding data to Solr is easy … once you have it the right format. For example, indexing information from a database is as simple as rendering the output of an SQL query into XML—something all modern relational databases can do. Even so, writing a crawler or connector is hard work for all but the most trivial applications. Crawlers have to know how to efficiently find relevant information in the repository, retrieve it, extract metadata and content, and send it to the indexer. In addition,

Table of Contents Feed for the Digital Edition of Enterprise Search Sourcebook 2008

Enterprise Search Sourcebook 2008 Contents Editor’s Note Publisher’s Note Findings and Figures Why Enterprise Search Will Never Be Google-y Searching for Search Usability Your Users Are Talking to You What’s Your Search Story? Search Is Dead—Now What? Delivering on the Promise of Enterprise Search Taming Multiple Search Engines in Your Organization Enterprise Search: Trends for 2008 Enterprise Search Seen From the Inside Open Source Search: Elixir or Poison? Avoiding the Big Mistakes in Search Semantic Search Takes Root in the Enterprise E-Discovery Essentials: The Rules You Need to Know SharePoint Search: An Enterprise Contender? Integrating Security Into Your Enterprise Search Infrastructure Engineering a Better Search Infrastructure Letting End Users Ask the Questions, Stat! The Power of Knowledge Legal Research Using Enterprise Search: A Developer’s View From Treading Water to Full Steam Ahead Pulling Out All the Stops With Midas A Natural Search Solution An Incremental Approach to Improving Enterprise Search The Enterprise Search Sourcebook Showcase Directory Index to Advertisers and Companies Mentioned

For optimal viewing of this digital publication, please enable JavaScript and then refresh the page. If you would like to try to load the digital publication without using Flash Player detection, please click here.