Expanding Web Search Part Two: Real Time Search

Yesterday I wrote about Focused Search-Based Applications as one way that Web and enterprise search is expanding. Here is another. The Wall Street Journal recently posted an interesting article on this topic, Internet Giants Look For Edge In Real-Time Search. It started by explaining that, “that real-time search helps Internet users find Web posts, including those from San Francisco's Twitter Inc., seconds after publication.” Twitter is one real-time search engine and others are joining the market.

The article goes on to add that making sense of real-time data is difficult for the big search players and their current algorithms. Their results are heavily weighed towards older Web pages that have established credibility and attracted large audiences, an approach at odds with real-time search.

Twitter users are constantly pouring out a flood of tweets, many are seconds old but from obscure users with little track record. How do you judge the quality and reputation that Google uses as part of its ranking? Also, tweets are written to take up as few characters as possible so spelling is often changed and acronyms, sometimes unique, are often used.

I see two different uses here. Google for time tested material and Twitter for what is happening right now. But how do you filter out the bad content in twitter like Google does? For now, the best way to determine the quality is to know the tweeter. On Twitter itself, Dion Hitchcliffe tweeted to me and @KMHobbie, “Twitter seems to be a solution at last for "filter failure." Rather have info endorsed by people I know.” I agree. I also look at the content, itself, and evaluate it on its own merits as most useful tweets point to expanded content in a blog or article. Meanwhile, there is a race to develop some relevant quality algorithms and spam filters for real time search.

I also asked the question here, When Do You Use Twitter Search vs. Google Web Search? I said that at times, I get better results on niche topics such as SOA or Agile development through Twitter search than through Google. I also get more current information.

Theron Kelso based on his own research added the useful comment: “Twitter search is better for: Current events, popular culture, and direct access to people for the purpose of continuing conversations.

Google is better for: current research findings (formal) in Science, Technology, Engineering, and Mathematics (with the possible exception of computer technology), how-to documents, and historical documents.”

Good points. However, the potentially fatal flaw for Twitter is that it dumps its data index after three months so you cannot search back after the ever rolling three month window. I use Twitter for personal knowledge management, just as I do this blog. I can go back and find stuff I blogged four years ago. The blog is a running record of what I feel is important. With Twitter I am saving useful links by tweeting or retweeting them. It is a great source to find information and to archive it for short term use. It complement sthe blog. I also tweet about events I attend but with the three emonth data index dump I cannot go back and find this information. So I am going to look for another real time search tool that can provide this, as long as they can migrate the traffic. Friendster lost followers to MySpace who lost some followers to Facebook. Will this happen to Twitter? You need to be able to go short and go long. Perhaps someone will build an archive app on top of Twitter just like TweetDeck works as an interface app.

So I discussed search-based applications and real-time search. Now where else will Web search go?

b

Comments

[Dennis McDonald]

Both the semantic and temporal distance between people and what they say is shorter on Twitter than on web pages indexed by Google. But it may be incorrect to refer to Twitter search as "real time"; "real real time search" would be the ability to identify people based on what they are doing (not just saying) right now. (Assuming they want to be identified, of course.)

[bill Ives]

Dennis - Do you mean an automated activity monitor rather than allowing people to say what they are doing? Some of the enterprise 2.0 tools do have automated activity monitoring status fields. Thanks for your comment. Bill