search (and destroy)

Yahoo! makes moves to acquire Overture, fueling it's growing search engine ambitions, while Google marches on and Microsoft is making "please-put-me-in-the-game-coach" faces on the sideline. Is there a search war a-brewing? And if so what will determine the victor? Read my extended entry for random musings on search engine competition and the next evolution of search engine technology: personalized search.

UPDATE (7/18): Here's a relevant Mercury News article with some more background: Search query returns reactions

Those of you who rank high on geekitude (if you need a metric, try the Geek Test - I scored 47.7%, making me an "Extreme Geek") already know that Yahoo!, still hot off its acquisition of Inktomi, is attempting to acquire Overture, who pioneered "pay-for-placement" search results and who earlier acquired AltaVista.

Strong currents are at work in the search engine world. While reigning champion and search engine super-hero Google continues it's strong march forward, Yahoo! is amassing it's own search engine super-powers. Meanwhile, in a dark cave in Redmond, Microsoft (or more precisely, the 50-member strong MSN search group) is gearing up to throw their hat into the ring. What does this mean? A showdown looms on the horizon.

But how would one hope to topple Google's page-ranking prowess? One avenue beyond leveraging the existing traffic and influence that Yahoo! and Microsoft (MSN) have would be to get the jump on the next technological advance in search technologies: personalized search.

When you enter a query at a search engine, it finds all the matching documents in the huge collections amassed by the engine's webcrawlers and then ranks them. What you see as a user are these results, presented in the order determined by this ranking. Part of Google's key to success was that its ranking mechanism, the now-famous PageRank algorithm (here's a research paper on it), exploits web hyperlink structure to identify authoritative pages on the web and rank them accordingly, creating results that you and I find much more relevant. Now imagine if such ranking mechanisms can be personalized for each and every user of the search engine. If the search engine has some knowledge of what your interests are (say, by reading in all your web bookmarks), it can tailor the result rankings to reflect those interests, resulting in search results that are that much more relevant to you.

One company has already tried this. Outride (formerly GroupFire) spun out of Xerox PARC (now PARC, Inc.) (actually, out of the very lab I work in!), with the goal of creating personalized search. Their technology was quite impressive, building off a number of PARC innovations, but they were in the right place at the wrong time. Sabotaged by the bursting dot-com bubble, and out in the market before search engine ranking performance reached the necessary threshold for robust, ubiquitous personalization, they failed to secure enough additional funding and had to shut their doors. As co-founder Jim Pitkow puts it, they were alternately "outridden" or "group-fired". Their assets (including all software and IP) were acquired by Google.

Fast forward to the present, where the clouds of search engine war may be starting to form. To achieve personalization, search engines will need to compute relevance rankings not once for the entire web (as currently done), but once per user for the entire web. Fortunately, students at Stanford's PageRank group have recently published papers on how to speed-up PageRank computation by using some clever linear algebra (remember Stanford has also birthed Google, and actually owns a fair chunk of them). Their resulting start-up company Kaltix (web page fairly bare, as they are in stealth mode) may indeed hold the key to making commercialized personalized web search a reality, which would make Kaltix the crown jewel in any search engine giant who may acquire them. Take your bets now.

And where am I in the midst of all this? Right in the middle it seems. The founder of Inktomi was my systems class professor, Outride was spun out of my own research group, multiple former Outride employees are now good friends of mine, and incidentally, one of the Kaltix founders is currently my housemate at my subletted summer residence. I'm excited to see what happens... I'll post more as things become fit for public consumption.