CMS Watch Endeca Feed

Cuil could be cool

bloem@radagio.com(Adriaan Bloem) — Fri, 1 Aug 2008 15:40:00 -0400

As the buzz has it, public website search engine Cuil is the new Google challenger. "Cuil" is apparently pronounced "cool", and "an old Irish word for knowledge".

The search engine was officially launched a few days ago and is enjoying its time in the spotlight. There's two reasons for that: the company was started by ex-Google employees; and it has an index that's supposed to be three times as large as Google's. Now, that's all very nice, but since CMS Watch doesn't evaluate the public search engines, but enterprise search tools ("behind the firewall search"), you may ask: what's the relevancy?

Well, the word is still out on Cuil's relevancy ranking -- or the freshness of its index, for that matter. One thing is certain: a larger index doesn't necessarily mean better results. The Cuil folks must have realized, though, that to be any kind of competition, your index has to be huge; it's the old numbers game that especially Yahoo! and Google used to engage in. Google was the first to quit playing that game, but somewhat "coincidentally" suddenly made a statement about their 1 trillion pages indexed.

As for how relevant this is for enterprise search: well, Cuil doesn't play that particular game (though many search companies do both or at least used to: Microsoft, FAST, Exalead, Vivisimo, the list goes on... and oh yes, Google). What struck me as most interesting is that Cuil attempts to change the way people don't just search, but find, by using an innovative new results interface. And that's always pretty good news... since so far, most vendors have rather unimaginatively been copying Google's design of search results, since that's what most users have grown used to on the web.

Of course, again, they're not the first to innovate: notable examples are the public search engines of Exalead (exalead.com) and Vivisimo (Clusty). Both are quite experimental, and especially Exalead is continuously updating the interface. What you like best is rather personal, but for me, both are more useful than Cuil, where a static footer on the bottom takes up too much of my screen real estate: frames are soooo 1996 (even if they're not actual HTML frames). But Exalead and Vivisimo's public search engines are more interesting because they are not just marketing, but also ongoing research: what you see there might actually turn up in an enterprise search interface near you soon.

Still, if Cuil will get people used to more varieties than just plain vanilla Google behind the firewall, as well, that would be nice for anyone trying to implement search. I think many would be quite happy to have users clamor for something that's more like Cuil, rather than "why can't we just have Google". It's time to innovate the interfaces beyond just Googlesque results listings and Endeca's facets. That wouldn't just be old Irish knowledge, it would actually be pretty cool.

Enterprise Search Vendor Landscape, Circa 2008

tregli@cmswatch.com(Theresa Regli and Adriaan Bloem) — Tue, 19 Feb 2008 00:16:00 -0500

You might be tempted to select enterprise search vendors for your shortlist based on their supposed "leadership" status in the market -- status either conferred by analyst firms or assumed by the vendors themselves. However, CMS Watch analyst Theresa Regli argues that you need to look more closely at product and vendor alike -- and understand where both are headed -- to properly evaluate your longterm risks and opportunities in an evolving marketplace...

The Enterprise Search Report 2008, updated, plus a Basic Search edition

tregli@cmswatch.com(Theresa Regli) — Thu, 14 Feb 2008 00:00:00 -0500

Today we release an update and a new edition of our Enterprise Search Report 2008. In addition to Microsoft's acquisition of FAST influencing several of our reviews, we've added 3 new product reviews, of SAP's NetWeaver Search, Open Text's Discovery Server, and Apache Lucene, bringing the total number of product reviews in the Enterprise Edition up to 20.

But for those of you who are looking to do a simpler enterprise search implementation on a smaller budget, we've also debuted a Basic Search edition (with Team Use and Intranet Site License options), with a smaller selection of vendors, but still all the selection and implementation advice you seek. You can compare the two editions here, and determine which fits your project scope and budget. We'll highlight more of our recent enterprise search research here in the coming weeks.

SAP invests in Endeca

tregli@cmswatch.com(Theresa Regli) — Wed, 6 Feb 2008 18:29:00 -0500

More quiet than the news of Microsoft's acquisition of FAST Search & Transfer was the recent announcement that enterprise search vendor Endeca has received $15 million in funding from the venture arms of Intel and SAP. As we'll point out in the upcoming update of our Enterprise Search Report, where we add an in-depth review of SAP's NetWeaver Enterprise Search, SAP has a legacy of its own search technology -- so why would they invest in someone else's?

Though $15 million is chump change for the likes of Intel and SAP, and Microsoft will have no problem writing the $1.2 billion check for FAST, acquisitions and investments speak volumes to the diversity of the technology in play, even within supposedly "niche" categories of products. SAP clearly doesn't see Endeca in the same target market, or addressing the same information findability problems, as their NetWeaver search product -- and neither should you. NetWeaver is tailored to search content that's part of a larger SAP platform; Endeca, on the other hand, focuses on more traditional enterprise search scenarios -- e-commerce, and website search in particular. Despite Microsoft claiming search is a core technology of MOSS 2007, they acquired FAST. Despite the debut of the new SAP NetWeaver Enterprise Search, an investment in Endeca is made. It only reinforces to you, the buyer and implementer, that one technology doesn't fit all. In fact, it seems Endeca is trying to convince Oracle they could also fill gaps for them that Oracle's SES doesn't currently fulfill.

Our suggestion to Endeca for the $15 million investment: how about adding search to your own web site?

Content cleanup in the former East Germany

tregli@cmswatch.com(Theresa Regli) — Wed, 26 Dec 2007 02:31:00 -0500

There's no time like the holidays for catching up on back issues of The Economist (don't worry, we're baking cookies, too), and this morning I found myself engrossed by a tale of pattern matching. No, not pattern matching of snowflakes or Christmas knits, but of a set of documents ripped into 600 million pieces by East Germany's State Security Service (better known as the Stasi), back when the Berlin wall was being torn down and the mob was at the gates. The Stasi were afraid of documents falling into the wrong hands, so when the shredders failed, they frantically resorted to tearing up documents piece by piece. And you thought getting your enterprise search engine to pull off late-binding security was tough?

In a project currently underway at Berlin's Fraunhofer Institute for Production Systems and Design Technology, software is being used to find patterns in these millions of Stasi-created fragments of paper and re-assemble them, jigsaw-puzzle style. In going through the fragments, the software is grouping the scanned shreds of paper together by identifying patterns in handwriting, color, paper texture, even ink color. Then, once a group of related shreds is found, the software puzzles the papers together. In their haste, the Stasi actually helped this process quite a bit -- most of the fragments of the same document were found in the same bag. Or bucket. Category. Taxonomy facet, if you will.

Like enterprise search tools that perform some sort of text mining and subsequent clustering -- such as Autonomy, FAST or Endeca -- this software has the capacity to learn and refine what it puts together, identifying new content as more or less like the original items in the set. When it gets confused (such as when a document has distorted or torn edges), it refers the act of judgement to a human being. But what's especially interesting about this software is that it actually spawns slightly altered versions of itself that compete for computer time on the basis of success at finding matches. Now that's something I'd love to see from my local enterpise search vendor.

There's a few lessons to be learned here. First, this is a multi-year project with dedicated resources, which is more than most companies are willing to commit to their own document scanning and indexing efforts. Second, while pattern matching may seem like an exact way to search for things, there's always factors in play that require judgement and refinement -- be it subtle linguistic differences, synonyms, or even how someone happened to tear something up.

And finally -- although history will surely welcome the Stasi's carelessness -- you should never take content security and storage lightly. You may think content is "secure enough," until you realize just how good your new enterprise search tool is at indexing all your content, but how bad it is at tying into your ACLs and showing the right results only to those who should see them.

Now, why can't I get my snowflake cookies to all look exactly alike?

Enterprise Search: Trends for 2008

bloem@radagio.com(Adriaan Bloem) — Thu, 29 Nov 2007 18:05:00 -0500

What's new in enterprise search? Very much and very little, argues CMS Watch Contributing Analyst Adriaan Bloem. Based on just-completed market research, Adriaan concludes that enterprise search customers and vendors alike are still grappling with key usability and technical challenges. But 2007 saw substantial marketplace ferment, and 2008 is likely to bring more...

Search the X-Files: unknown entities

bloem@radagio.com(Adriaan Bloem) — Fri, 21 Sep 2007 07:46:00 -0400

If you're in the market for search technology, you probably hear a lot about faceted browsing, guided navigation, refining, clustering, categorization, and so on. Many of today's search engines attempt to present more than just keyword search. That's fine if your content has high-quality structured metadata, but what if you throw in thousands of Word documents where the "author" is defined as John Doe? The truth may be out there -- but the answer is buried deep in the text.

Distilling things like people, email addresses, and company names from source content is what is known as "entity extraction." Vendors may tell you that yes, their search interface pivots off that kind of data (e.g., for guided navigation), but don't worry: they can extract the unknown entities even if you throw in large files nobody ever bothered to tag right. Enterprise search will create and then reveal structure where once there was chaos.

Of course, this is not at all the black magic it is made out to be. Finding relevant entities is usually accomplished through a combination of pattern-matching and dictionaries. An email address will contain the "@" symbol, and it's pretty safe to say that if it's followed by a dotted domain name, you've got your address. If "John" is in your dictionary of first names, the next capitalized word will probably be the surname. This also means that entity extraction is language- and even country-specific. A representative of Fast Search & Transfer's professional services told me about the challenges the company faced finding a fail-safe way of distilling German street addresses, which have a very different and much less formal structure than those in, say, North America.

Many vendors, of course, won't like you to be distracted with the details of their "automagical" ways of achieving this. Their method may be English- and US-specific, but hey, so what -- if your company is based in the US and content comes in English, you're fine. In reality, things are never that easy though.

I was running a test of ISYS:web against the CMS Watch website, and was pleasantly surprised to see the out-of-the-box installation correctly identified several countries, and had no problems finding out that Tony Byrne is an actual person. It even managed to extract Janus Boye's somewhat more exotic Danish name. Unsurprisingly, Apoorv Durga was a bit too outlandish and my ego wasn't hurt when Adriaan Bloem wasn't ranked among the people. But you really don't want to provide Theresa Regli with cannon fodder by ignoring her (which it did), while on the other hand, I can't recall ever having met "Read More," international man of mystery, now a full-fledged person in my search engine.

This is not to say you should bash ISYS for this -- the company is the first to admit its methods aren't infallible, and many vendors at a much higher price point don't even offer similar technology, instead relying on third-party tools. What it does mean, however, is you shouldn't take claims that "it's all taken care of" at face value. Investigate whether languages and countries relevant to you are supported, and better still, test against your own content. Then assume you are committing yourself to near constant system training and tweaking.

Failing that, some search products will allow you to specify additional criteria (with ISYS, for instance, "Theresa" was easily added with a [pre] construct in a text-based configuration file). Others enable you to define completely new entities and patterns from scratch (such as FAST's processing in Python, or Endeca's XSL and Perl). Be very aware, though, that sending in a Mulder agent to investigate your X-Files might be a costly, ongoing adventure, lasting nine seasons of suspense.

Coming next...a patent on hit highlighting?

tbyrne@cmswatch.com(Tony Byrne) — Fri, 2 Jun 2006 13:52:00 -0400

Here we go again. Search vendor Endeca claims a patent on "guided navigation." I've written previously about various patent absurdities in our space (Interwoven, Vignette, Oracle, 24/7 Media). Fortunately, the faceted browsing genie is out of the bottle, so I don't think customers need to care. As our new search report points out, nearly every vendor now offers guided navigation of some kind or another. At its core, this just means ordering and counting resultsets according to attributes, and then making drill-downs available to search, order, and count available subsets. Epicurious does this famously well. Some web applications do the same thing with simple SQL queries; that approach doesn't scale very well, but you get the idea: guided navigation is not new. A deeper read of Endeca's patent seems to suggest an emphasis on their particular approach, including their take on text pre-processing (i.e., automated metadata generation at index-time). Hopefully, the company will just wave the patent around to investors.

Oracle gets sexier with TripleHop acquisition

tbyrne@cmswatch.com(Tony Byrne) — Thu, 15 Sep 2005 15:26:00 -0400

While I wasn't looking this past summer, Oracle "acquired certain assets" of TripleHop, a specialized enterprise search vendor. Oracle didn't make much noise about it at the time (e.g. no press release), but it looks like TripleHop customers and staff are both moving over. As Enterprise Search Report readers know, TripleHop's technology offers Oracle many new-fangled (if not well battle-tested) techniques, like contextual search, as well as specific industry applications, to complement Oracle's traditional emphasis on powerful search infrastructure. Oracle's pitch to search customers has tended to emphasize core engineering and (usually impressive) test results. Now look for Oracle to try to challenge Endeca and other pure-play enterprise search vendors head-to-head on features.