CMS Watch Apache Feed

How Fast is Attivio?

bloem@radagio.com(Adriaan Bloem) — Sat, 5 Jul 2008 09:55:00 -0400

The Fast Search & Transfer news continues to keep me on my toes. The Norwegian business weekly Dagens Næringsliv seems to have come up with some decent evidence (English translation) of many things everybody already suspected -- and a couple of new ones.

It covers the Convera acquisition and the "separate" but surprisingly coincidental deal where Convera bought several million's worth of Fast software it didn't need. Or as rival Autonomy (which was also in the running for buying Convera) has pointed out, Fast pumped up its revenues for that quarter with part of the money it paid for Convera, then got back for licenses. The DN article also covers a few other very suspicious deals, and some outright fraud. It's now even getting to the point where calling Fast "the Enron of Norway" is getting long in the tooth.

While that train wreck was unfolding before my eyes in slow motion, my fellow analyst Theresa Regli pinged me last February about a new enterprise search company called Attivio. Information Today raved about their new product AIE, with analysts quoted as saying things like "they are moving rapidly to develop tools that will eliminate many of the practical barriers to easily and efficiently deploy robust enterprise search solutions," with the unique selling point of "data integration plus search and content processing," a "hot niche for the next few years."

Since I'm always interested to find out more about robust enterprise search tools to fill hot niches for the next few years, I scrolled down to read what the Attivio CTO would explain about how the product would achieve what "should have been solved by the integration of text search and XML into relational database managers such as Oracle." As it turns out, it is based on a "mash-up" of open source Apache Lucene and "licensed commercial software."

As described in our Enterprise Search Report and on this blog, Lucene itself is just a Java text search API. To be able to actually gather, convert, and query content you need many more components. It is perfectly feasible to put together a working enterprise search product around the core Lucene JAR (as demonstrated by IBM's Omnifind Yahoo! Edition). But in order to get there, and to have Lucene index, for instance, Office documents and PDFs, you will have to first convert those documents to text. The filters to perform that conversion can be bought from other vendors, based on open source such as pdftohtml, or you'll have to build them yourself, which is a lot of work. There aren't too many vendors building their own filters, or even just modifying open source to do so. So if you do build the filters needed to use Lucene yourself, you'd probably like to mention this as an advantage, and as Attivio states, "we developed our own Microsoft Office, WordPerfect, and PDF connectors to improve performance and reach deeper into the files than the conventional converters."

Since, like most enterprise search products, Lucene isn't based on a database and couldn't even connect to such content without help, it isn't surprising Attivio had to develop a "unique RDBMS data loader" which "indexes the tables individually." This, again, is presented as a major advantage -- remember, converting documents and integrating structured and unstructured data are "a hot niche."

I remember seeing a vendor at a conference a few years back, with banners jokingly stating its product was "buzzword compliant!" Attivio certainly seems to have that skill down. The engineering effort is marketed as a "technology mashup," "breaking down silos" between "open source and commercial software." "We have lived with the challenge of having to choose between the precision of databases and the richness of search for a long time, but no longer" sounds great, but I don't see Oracle and Thunderstone's RDBMS-based solutions breaking out in a sweat just yet.

Maybe my over-exposure to marketing materials and flashy demos has turned me into a cynic, and Attivio's downloadable trial version will have to do at least a decent job to convince me of the product's added value. Fortunately, that free download is "coming soon!" Yes, I'm sorry, I'm finding it increasingly hard to turn off that cynicism, especially when I turn back to the DN article about Fast Search & Transfer. Attivio was founded by former Fast employees and the Attivio CTO is Sid Probstein, formerly vice president of technology at FAST. More importantly, Attivio's CEO is Ali Riaz, who was COO at Fast but unexpectedly left the company in late 2006. Well, in hindsight, perhaps not so unexpectedly, though DN quotes him as saying "I had nothing to gain from manipulation of the accounts. I had no shares in the company. I wanted shares and quit because I didn't get any. If you want to find out what's wrong with the accounts, you need to look at those who could gain from it. And it wasn't me."

Dagens Næringsliv doesn't appear to agree with Riaz, however; if you want the full analysis of the what and why, I suggest you read the article. I myself find it surprising that the CEO of a technology startup backed by $6.2 million in venture capital would drive an Audi R8, but that doesn't mean anything (other than that I'm envious of his car). I also find it surprising a former Fast COO would be co-owner of a company reselling Fast licenses, but walking like an Enron duck and quacking like an Enron duck doesn't necessarily mean that it's really anything like Enron. And Attivio's clouding the core technology in marketing hyperboles and buzzword compliance is slightly disconcerting, but many renowned companies engage in the same practice.

DN quotes Riaz as saying "you should be much better at praising the people who have success, instead of pushing them down." And I would certainly love to be proven wrong by Attivio's software; as soon as I get my hands on the trial download I requested, I will let you know if it lives up to the high expectations. As one of my teachers in school once told me, "I'm known for being cynical, or even sarcastic -- myself, I prefer to call it healthy skepticism and mild irony."

Being a cynic isn't a lot of fun -- but for now, I would advise you to be at least healthily skeptical of what Attivio has to offer.

Enterprise search: free as in free beer?

bloem@radagio.com(Adriaan Bloem) — Fri, 16 May 2008 13:23:00 -0400

Searching information -- really, how hard can it be? So, why wouldn't you go out and get a search engine that's for free? Well, to stick to the analogy of "free beer," you might wake up in the morning with a headache, only to find your wallet gone.

Of course, I'm paraphrasing the definition of "free software". Richard Stallman's example is used to point out the ambiguity of the term "free" in the English language. With free software, "you should think of free as in free speech, not as in free beer." Nevertheless, you should be warned: both open source beer (now in version 3.3) and free commercial beer have the potential for leaving you with a bit of a hangover.

If you really think enterprise search is a simple commodity -- and I will only comment on that with the obligatory statement that readers of our Enterprise Search Report will probably know better than that -- getting a free product would be ideal to get your feet wet (albeit somewhat sticky). I get invited to BYOB enterprise search parties a lot, and usually come up with Apache Lucene, IBM Omnifind Yahoo! Edition, and Microsoft Search Server 2008 Express. Let's get a closer taste of each.

Apache Lucene. Lucene is open source, which you are free to use. The problem is, it's not a complete enterprise search product -- it's a "text search engine API." What you get is a Java JAR with the core functionality of a search engine. In typical hardcore Java developer understatement this is described as "you write the easy stuff, the UI and the process of selecting and parsing your data files to pump them into the search engine, yourself." To developers that doesn't sound too difficult -- it's a library they'd be able to use to create search functionality for many applications. As they embark on that journey, however, many will find out they'll have to become experts on enterprise search to get their implementation to perform basic tasks any Google user has come to expect. Index Word documents? You'll have to convert those to text first. Remove stop words or perform spell checking? You'll have to get some more jars to fit that in. And that familiar user interface isn't so easy to replicate, either.

Of course, there's a couple of more "pre-packaged," Lucene-based engines (such as Nutch and Solr), but they'll only take you so far on that long and winding road. There's some excellent examples of what you can achieve with Lucene, but many more of how hard it can be to get there.

IBM Omnifind Yahoo! Edition (or OY!E). The Google appliances have the Google brand behind them, which must have got the IBM people thinking the Yahoo! brand would be excellent marketing for their free-to-use search engine. In fact, it's neither IBM nor Yahoo's technology, but Lucene wrapped in other open source software. A few commercial bits thrown in create a product that's easy to install and run. It will actually do many of the things Lucene will make you work hard to accomplish: it comes with support for several languages and quite a few source content filters. For users, it looks like a regular web search engine; for admins, there's a nicely designed and intelligible interface. In short, it does most of the things a Google Mini appliance will do -- but for free.

So what's the catch? Well, the license (by the way, what license?) limits you to 500,000 documents and 5 collections. After that, you can "upgrade" to other Omnifind products. But since the technology across the Omnifind line-up is completely different, this is the same as starting from scratch, and you'll pay for the privilege. I've been critical of the limitations of Google's appliances in the past, and sure, the 50,000 document limit of the entry-level Google Mini is a lot less than OY!E's half a million. But that comparison isn't really fair, considering the fact the Mini actually comes with the hardware to run the queries on for a mere $2,990. And don't think you'll be able to run IBM's software on an old abandoned test server you have available -- OY!E will need more power than the single blade Google Mini or Thunderstone Appliance to match the performance. Tellingly, I wasn't able to dig up an example of an OY!E implementation to mention while researching the Enterprise Search Report (if you know of one, let me know).

Microsoft Search Server 2008 Express. Microsoft's free offering is basically the same software as the non-Express version, but then there's the seemingly innocent limitation: one server only. I wouldn't want to continue the theme of this post by saying this is akin to handing out free samples of beer to get you hooked; suffice it to say that if you start to run the Express version in a production environment, there will, no doubt, come a time when a single server won't be enough anymore. When you've come to rely on the solution, you'll suddenly have to shell out for the licenses. As I've said before, having a free lunch isn't necessarily a bad thing; just remember that you'll probably have to pay for the beer the lunch comes with.

So, this might all start sounding like advice your mother gave you: never take anything from a stranger, and certainly no free alcoholic beverages. Don't forget, however, that I'm Dutch, and I've certainly developed a taste for enterprise search. Free beer sounds too good to be true, but it could certainly get your party started; just remember to drink in moderation, and never, ever, drink and drive.

Enterprise Search Vendor Landscape, Circa 2008

tregli@cmswatch.com(Theresa Regli and Adriaan Bloem) — Tue, 19 Feb 2008 00:16:00 -0500

You might be tempted to select enterprise search vendors for your shortlist based on their supposed "leadership" status in the market -- status either conferred by analyst firms or assumed by the vendors themselves. However, CMS Watch analyst Theresa Regli argues that you need to look more closely at product and vendor alike -- and understand where both are headed -- to properly evaluate your longterm risks and opportunities in an evolving marketplace...

The Enterprise Search Report 2008, updated, plus a Basic Search edition

tregli@cmswatch.com(Theresa Regli) — Thu, 14 Feb 2008 00:00:00 -0500

Today we release an update and a new edition of our Enterprise Search Report 2008. In addition to Microsoft's acquisition of FAST influencing several of our reviews, we've added 3 new product reviews, of SAP's NetWeaver Search, Open Text's Discovery Server, and Apache Lucene, bringing the total number of product reviews in the Enterprise Edition up to 20.

But for those of you who are looking to do a simpler enterprise search implementation on a smaller budget, we've also debuted a Basic Search edition (with Team Use and Intranet Site License options), with a smaller selection of vendors, but still all the selection and implementation advice you seek. You can compare the two editions here, and determine which fits your project scope and budget. We'll highlight more of our recent enterprise search research here in the coming weeks.

Enterprise Portal Marketplace: 2008 Vendor Risk Profile

jb@boyeit.dk(Janus Boye) — Thu, 17 Jan 2008 00:01:00 -0500

It is all too easy to identify vendors for your shortlist based on their supposed "leadership" status in the market. But CMS Watch contributing analyst Janus Boye argues that CIOs, procurement officers, and other technology leaders considering investments in enterprise portals should carefully examine the risk profile of prospective vendors to help identify the right "fit" for their needs.

Apache Shindig: where does the portal end and the social application start?

info@jboye.dk(Janus Boye) — Thu, 20 Dec 2007 19:45:00 -0500

Last month, Apache member Brian McAllister proposed a new incubation project called Shindig, which would create an open source implementation of OpenSocial. OpenSocial is Google & Co's much hyped response to Facebook, offering a new common set of APIs for social applications across multiple websites.

To quote the interesting Shindig proposal:

A social application, in this context, is an application run by a third party provider and embedded in a web page, or web application, which consumes services provided by the container and by the application host. This is very similar to Portal/Portlet technology, but is based on client-side compositing, rather than server.

Perhaps as a surprise to some, in particular outsiders to Apache, Shindig and Jetspeed (the official Apache Portal Project) are not connected in any way. For Apache this is the usual way of working: innovative, but often disconnected projects sometimes overlap or supercede each other. As another example, Jetspeed does not use Lenya, the Apache open source content management system. Rather Jetspeed works closely with Hippo, a Dutch open source Web CMS vendor.

Shindig could end up as the open source reference implementation for OpenSocial, but it is still very early days. Jetspeed has certainly not been as universally adopted as the famous Apache webserver. As an open source adherent, you should not assume a coordinated master plan among Apache projects and definitely not count automatic universal adoption for any tool that has the Apache moniker. At the same time, let's recognize that Shindig looks quite interesting, at the very least.

Thanks for Navaneeth over on The Portal Zone for picking up the story.

Scalability the Terracotta Way

kthomas@cmswatch.com(Kas Thomas) — Tue, 14 Aug 2007 06:23:00 -0400

One of the theoretical advantages of Java-based Portals and Content Management applications is the ability to cluster servers for better performance. But the reality is that clustering is a black art that few vendors and implementation teams really ever seem to master adequately. So it comes as a (welcome) surprise to learn of an open-source technology that delivers many (if not most) of the things customers want here, but in surprisingly quick, painless fashion, at low cost, with no need to recompile code or stay up nights learning about disturbing-sounding concepts like "STONITH" (shoot the other node in the head).

The technology in question is called Terracotta, and it works by clustering the Java Virtual Machine in such a way that even a participating JVM itself doesn't know that it has been enlisted in a coordinated effort of any kind. Through a clever bit of boot-time dependency injection, Terracotta patches a handful of core JVM memory-management bytecode instructions, achieving transparent virtualization across any number of enlisted VMs, under the control of a Terracotta server that lives in "aspect space." The Java memory model is not altered. Application code does not have to handle locks any differently or follow any special APIs, or even know that it's been clustered. Have I lost you here? Think of it this way: Instead of implementing special cluster services at the application level using product-specific APIs, Terracotta clusters the Java heap itself, underneath your applications.

It all sounds like science fiction until you try the tutorials, read the white papers and technical literature, and examine the long list of integration efforts (listed on the Terracotta website) involving other Java-based modules like Apache Lucene.

One of the more intriguing integration efforts thus far has been Geert Bevin's recent quest to achieve heretofore unknown levels of scalability and performance with the open-source Web CMS package, Drupal. Drupal is actually written in PHP, but in this case runs on Caucho's Quercus (a Java implementation of PHP), leveraging Terracotta in the cache layer. As Web CMS Report readers know, Drupal is a collaboration-intensive CMS solution of the "let's cache everything in the database" variety -- with difficult scalability problems to match. Bevin's system is highly experimental at this point, but it hints at what people might be able to accomplish with the technology.

In the meantime, other content technologies that take advantage of well-known Java subsystems like Hibernate, Tomcat, Resin, EHCache, Quartz, and so on have the most to gain by exploring Terracotta as a fast path to scalability. Individual subsystems can be tested against Terracotta separately, to find sweet spots.

It will be interesting to see how long it takes mainline ECM and Portal players (particularly those that rely heavily on Java-based infrastructure components) to include Terracotta in their "supported product configurations." I would expect the Alfrescos and Liferays of the world to stay out in front of the situation. Purveyors of complex proprietary solutions might miss the boat.

Scalability always has been (and probably always will be) the Achilles' heel of all the technologies we cover. I'll be watching to see how other communities adapt Terracotta-like notions to other well-known virtual machines (e.g., .NET). Anyone at www.mono-project.com listening?

IBM and Yahoo to Offer Free Enterprise Search Engine

apoorvdurga@gmail.com(Apoorv Durga) — Thu, 14 Dec 2006 05:42:00 -0500

IBM and Yahoo have announced a free enterprise search software. Called IBM OmniFind Yahoo! Edition, it neither uses IBM's nor Yahoo's technology for underlying search but instead runs on Lucene, an open source search technology from Apache. Since Lucene is already popular option for implementing search for many J2EE implementations, this could present a good option as it should be easier to integrate and extend it without learning new proprietary search APIs. This promises a good entry level solution that is easy to setup and configure -- qualities generally not associated with search products. So it'll be interesting to see how other search vendors respond. Yahoo's contribution is its familiar search interface -- yet another example of how consumer web technologies are making inroads into the enterprise.

CMS Watch updates Enterprise Portals Report

tbyrne@cmswatch.com(Tony Byrne) — Tue, 10 Oct 2006 00:29:00 -0400

Today we release the 2nd edition of the Enterprise Portals Report. In this version we begin coverage of the eXo and Jetspeed portal platforms, and have added new scenarios and a longer review of business process management (BPM) and business intelligence (BI) features. Most importantly, all vendor evaluations have been updated, including analysis of the latest portal offerings from IBM, Microsoft, Oracle, and many more. Read the full press release. As always, you can download a free sample.

Jetspeed 2.0 introduces bridges to other development frameworks

info@jboye.dk(Janus Boye) — Thu, 15 Dec 2005 15:13:00 -0500

The Apache Portals Jetspeed Team recently unveiled Jetspeed 2.0. An interesting new subproject is called "Portals-bridges" which includes common code and demos for the use of common web frameworks via portlets. This means you can write portlets using JSF, Struts, perl CGI scripts, php code and Velocity templates. Also noteworthy in the new Jetspeed release is a new component architecture and full support of the JSR-168 portlet specification. Jetspeed-1 users should note that this is a complete rewrite from the ground up, and little may be familiar under the hood.