Portals and KM

Why I Still Like Text Over Slides and Video on the Web

I have written on this blog and elsewhere that the phonetic alphabet was the greatest ever breakthrough in information technology. I still think is the best medium for the Web. Now I would add that it is best supplemented with images but images without text frustrates me. When I get a link to a slide share or a video I frequently do not bother the look at it. Here are some reasons.

You cannot grab quotes or paste into another doc.

Text is very scannable – according to Jakob Nielsen users have time to read at most 28% of the words during an average site visit and 20% is more likely but it is easier to come back with text.

Text is easy to distribute – It’s easy to cut and paste it and send it somewhere or to clip and re-syndicate it via email, RSS or social networks

Text is easy on the workplace – It’s much easier for cube-based workers to read text on the screen and get away with it vs. watching long videos.

Search results are still largely made up of text and better SEO is good thing if you want to get your message out.

Now the ancient Greeks who came up with the phonetic alphabet probably did not think about these implications. It is the invention that keeps on giving. The concept of an original version that could be preserved did not evolve until after written text. This was critical to the development of modern science among other things. In many ways, the epic poets, chief knowledge distributors of their day, made up the details as they went along. Text made available a visual record of thought, abolishing the need for an acoustic record and hence the need for rhythms. The first thing put into text was an epic poem but soon text inspired documents appeared.  But there is more.

Like many great inventions, the uses of text slowly evolved. In fact, it appears that reading was often done aloud until after the 6th century. Ivan Illich relates that St. Augustine refrained from reading after his brothers went to sleep for fear of waking them. After the 6th century, silent reading became more commonplace, and such techniques such as tables of contents and indexes first appeared over a thousand years after the phonetic alphabet first emerged. These new devices allowed for random access to text information, a concept we take for granted now.  Now people were free to read their emails and blog posts without disturbing their neighbors. What will be next for text?

 

 

Competitive Intelligence System Success Factors

I remain very connected to Darwin Ecosystem and follow their blog. One post offered some advice I wanted to expand on here. Nicholas Herold wrote the post, 8 Qualities of a Successful Competitive Intelligence System. He began by pointing out the fragmented way many organizations go about competitive intelligence.  Even need a competitive intelligence system is used there can still be issues of finding the right stuff at the right time and noticing the outliers that are predictors of critical market shifts. Nicholas lists eight qualities that you should look for in a competitive intelligence systems to reduce the risk of missing what is truly important. They are found below with some comments added.

1. Monitor unique topics of interest – you need to be adjust what you are looking and focus in on clearly defined areas to discover what is happening with precision

2. Operate in real time – this is critical, otherwise you will be a follower, rather than a leader.

3. Be independent of sources – you need to be able to pull in multiple sources and not  be tied to a few that n might bias what you art e looking at

4. Be human centered (rather than allowing algorithms to determine relevance and importance, the experience and knowledge of the human is utilized),=. This is one of the most important. Tools like Google decide for you what is important and arrange content according to their viewpoint.  The person needs to have clear visualization of what is happening so they can scan it and make intelligence decisions on what to drill down on.

5. Eliminate the noise – you need filters to get rid of unrelated content

6. Display emerging patterns – good visualizations of changes in the topics of interest helps you discover trends that may impact your decisions

7. Simplicity – allows for the human to use the tool effectively. Many complex competitive intelligence tools require too much expertise in the tool, putting distance between them and the expert on the content of interest.

8. Speed – this is enabled by both simplicity and real time content aggregation

These all capabilities we build into the Darwin Awareness Engine™ For information about how Darwin can improve your competitive intelligence system, see our CI White Paper: Darwin Awareness Engine and Competitive Intelligence.

 

Search vs Discovery – Finding Useful Content

Search and discovery are both useful ways to find useful content. Everyone knows about search. Google is my home page. It has opened up the Web to the world. As Romain Goday wrote in Search vs Discovery (Or A New Approach to Information Consumption), “Search has boomed in the last decade, more than 130 billion searches were conducted during 2010.”

While search provides an answer to a question you already know. “Discovery” provides awareness to things you might not have anticipated in areas that interest you. While still at an early stage, there is a new breed of tools, content discovery engines, that are gaining traction to complement search engines. They do not replace search they provide another line of inquiry into what is happening on the Web.

Discovery goes beyond search as it can help you uncover meaningful content without having to formulate a question.  It can provide ongoing monitoring of an area of interest and report back variances in information patterns. It operates in real-time and does not have to wait for popularity to drive content forward. This is approach taken by the Darwin Awareness Engine™, one of the new breed of content discovery tools. 

 

Bing Moves Past Google on the Social Side of Search

I have been writing a bit about the invasion of social capabilities into all online tools. Fairly soon this will no longer be news. I remember in 2004 bloggers were excited every time some big shot started blogging. Now that is a yawer.  Now we are reading about how the Weather Channel is going social and how LinkedIn value dramatically increased after it introduced social features. Of course social business is the big headline in enterprise tools but social search appears to be still in its infancy

Now the MIT Technology Review writes about: With New Look, Bing Gets More Organized, Social. They announce that Bing will transform its interface a layout in three separate panes featuring traditional text results, results from specific information sources and services, and results related to your social-network friends. Sounds a little like Tweetdeck meets search.

I have heard from some Bing executives a while back that they were going to make the search tool more social and they seem to have pulled it off. Now we will see how it affects Google's 66.4 percent lead over Bing’s 15.3 percent of U.S. search queries in March, according to comScore.

The MIT articles also notes that the new capabilities will go beyond the new interface. For example, Bing will now suggest friends who might be knowledgeable about a specific topic by considering their listed "likes" on Facebook. A feed will also let you see and respond to your friends' Facebook updates and questions. And users will be able to ask their Facebook friends questions via Bing.

There is the rule proposed by Andy McAfee and others that the new tool needs to be ten times better than the current one people use. We will see if all this social stuff meets the 10x rule.

 

 

Information Overload Can be a Misleading Term

It have been a while since I wrote about the Darwin Awareness Engine™ but it is alive and doing well gaining new customers. Nicholas Herold makes a nice point in the post, Information Overload? Or Is it just noise?  The Darwin Awareness engine does not help with those simply overwhelmed with too much information. Instead it organizes information into patterns that help the knowledgeable person make sense of what is happening in their area of interest.

The concept of massive amounts of information does come into play but it is not too much information if you can sort it into groups that make sense and then efficiently explore those groups to find the outliers that offer meaningful insights. In this sense having massive amounts of information is a good thing if you can process it effectively. That is what the Awareness Engine is designed to do.

Content becomes overwhelming if you do not know what to do with it. But if you know you subject, the more content the better if you have the right tool to quickly go through it to find content of value.

 

 

Introduction to Google Analytics – Online Workshop

Google Analytics is a popular free tool from Google that provides a wealth of information about the visitors to your website. It can take a bit of time to uncover its full use. I want to alert you to a useful online workshop offered by my friends at Pioneer Training. As a disclosure I have no business connection with this firm except that I have been a close personal of the owner, Don Lesser, for years and have written about his food writing on this blog before. I normally do not offer such promotions but I see this as a real service for an important product. Readers of this blog please do not send me similar requests to promote your products.

This two-hour online workshop will introduce you to the benefits of Google Analytics. This online class uses GoToMeeting software.  You will be prompted to download a free guest copy prior to the class, if necessary. (Installation is quick and painless.)

Use Google Analytics to find out:

• Number of visitors
• Average time visitors spend on your site
• Types of traffic (direct, referral, search engine)
• Keywords used to find your site
• Most and least popular pages
• Geographic location of visitors
• Number of “socially engaged” visitors (Facebook, Twitter, Linked In)

Set Goals and Track Visitor Progress

• Measures conversions from visitors to customers (or donors, volunteers, etc.)
• Track visitor progress through your “funnel path” to a goal.
• Use data to improve site content and design.

Real-Time Data and Custom Reports

• Find out how many visitors are on your website now!
• Reveal the key words they used to get there.
• Show the pages currently being viewed and locations of current visitors.
• Customize reports for focused data analysis.

Regitser Online, by Phone or E-Mail

• Cost: $99.
• Go to their website: http://www.ptraining.com.com.  Click Register Now! on the home page.
• Call their office: (413) 387-1040
• Send them an email:  [email protected]
• PPrivate classes using your website or blog data also available.

Private SEO website analysis also available

• Learn how to optimize your website for search engines.
• Improve your SERP (Search Engine Results Pages) rankings.
• Avoid SEO mistakes and make maximum use of web page copy, titles and ALT tags.
• Design Google Analytics goal funnels to track visitor progress through your site.
• Create custom GA reports to retrieve the exact information you need

 

Forrester's Leslie Owens Covers Enterprise Search

Last month I attended Forrester's Content & Collaboration Forum 2011. The Forum explored what the “current demand for more portable, social workplace experiences means for your workplace strategy.” It shared the “latest trends in technology adoption and how firms forge better business outcomes from a more mobile, social, and virtual workforce.” I had a chance to speak one on one with Leslie Owens. She spoke at this event the next day and here are my session notes (Forrester's Leslie Owens on Harnessing the Voice of the Employee).  (Leslie’s expertise includes: enterprise search, semantic tools, information classification, and taxonomies and I have covered her work before (see for example, Forrester on Enterprise Search Trends).

We first discussed looking into the voice of the employee, the topic covered in her talk and recorded in my session notes above. I will not repeat the parts she covered in her talk. Leslie noted in our conversation that the enterprise is a bounded community. You know all about the participants who engage in conversations within the enterprise social media platforms. You know much more about them than the customers or prospects who participate in social media on the Web. I think that many of the applications of social media such as wikis and activity streams can have even greater value within trusted communities within the enterprise. In this case I am not suggesting that listening to the voice of the employee is of higher value than the listening to the voice of the customer. They complement each other. On some issues employee conversations may help better explain what is being discussed by customers and visa versa.

We also discussed the privacy issue a bit more. People objected to data mining emails because the expectation was that they are private.  Personally, I think this is mistaken assumption. They are certainly discoverable in a legal action. It is my understanding that most company email systems indicate that they can be monitored for inappropriate actions. However, emails are not immediately transparent to everyone so there is some expectation for privacy.

With social computing inside the enterprise the expectation is transparency. It is actually seen as one of the benefits. So looking at the voice of employee with better analytic tools is simply doing a better job of maximizing this opportunity and increasing the return on investment in social tools. It is doing what they are designed to do. People still need a secure place to communicate but permission levels can be established. There are also other channels outside social media. Privacy is important and there is too much invasion of it today but enterprise social media is the place for transparency within a trusted group, either the whole enterprise or a subgroup within in it.

We also briefly touched on some of the current trends in enterprise search. Call center support is one area of innovation. Custom searches for call center agents bring in relevant content from diverse sources. This is a much-needed improvement. I have done a lot of work with knowledge support for call centers. One agent in the UK said to me that one of the most important skills was the ability to engage the caller in amusing chat will you frantically searched through multiple databases for the answers to their questions.

This unified search is also extended to customer facing web sites (see for example; Coveo Customer Information Access Solutions (CIAS) Moves to Version 2.0). Here is an excellent report that Leslie and her colleagues did, Site Search Evolves from Technical Feature to Customer Touchpoint, and here is my commentary on the report - Evolution of Site Search.

There is also much work being done in federated search going across structured and unstructured data. Here is a conversation I had with Sid Probstein, CTO at Attivio, a firm focused on this issue. I started by asking Sid about the opportunity within enterprise 2.0 to take advantage of looking at the unstructured data to determine the pulse of the enterprise. Sid responded and also showed me a demo of how Attivio can pull unstructured content from product review sites on the Web and index it along with structured data from a CRM system. Here is an earlier review of their work (Attivio Tightly Integrates Structured Data and Unstructured Content for a New Approach to Information Access).

Leslie added that enterprise search is behind Web search on a number of issues beyond the voice of the customer/employee issue. Personalization and making effective use of past search behavior are areas where enterprise needs to catch up. There are great opportunities to provide more individual relevance within the enterprise. The rise of enterprise social media should help accommodate this need by providing more context to aid search engines, just as it is doing on the Web.

 

My KM World Session Notes: Creating a Semantic Infrastructure Platform

I am primarily posting my notes from the 2011 KM World on the Darwin blog. I will share a key sessions here and then provide a summary of all my notes with links. I attended the workshop – Creating & Applying a Semantic Infrastructure Platform for Knowledge Sharing led by Tom Reamy, Chief Knowledge Architect - KAPS Group. Here is the session description.

While the distinction between information and knowledge has often been misused, there is an essential truth that knowledge is richer and deeper than simple information. In the early days of KM, there were numerous attempts to utilize information technologies and approaches (search, taxonomies, etc.) for knowledge sharing, and the results were not pretty. However, today, with a rich array of text analytics and knowledge platform software, we are in a position to support real knowledge creation and sharing, but only if we approach it with the right foundation. That foundation is based on a concept of a semantic infrastructure, an infrastructure needed to support all the ways in which people in organizations use language and meaning. This workshop explores the essential characteristics of creating, refining, and applying a semantic infrastructure for rich knowledge sharing. It starts with how to do a knowledge audit designed to create a platform for knowledge networks that provides the means for enhanced communication within and among self-defined social groups. It then looks at a range of text analytics tools including text mining, auto-categorization, entity and fact extraction, summarization, sentiment analysis, and how these tools can now go beyond mere information applications to form the basis of a range of knowledge-sharing apps.

Tom began by stating that he would first define semantic infrastructure. He said that then he would cover implementation issues and benefits and get to tools later. That is a good order. He asked how many people were involved in KM and everyone raised their hands. Tom is a consultant in this space and has a virtual group that works with him.

Semantic infrastructure has four dimensions. Ideas and content structure is the first. You need to look at all the content, its structure and how it is used. People are the next dimension as creators and consumers of content. Activities are the third dimension. This means the information behaviors associated with the work process more than the process itself. Technology is the fourth dimension.

With content and content structure you need to look at both structured and unstructured content, both inside and outside the organization. It also needs to include the tacit knowledge inside people’s heads. You need to model flexible schemes to handle all this stuff. It includes faceted metadata, simple taxonomies with intelligence (such as an auto-categorization tool as taxonomies can be static otherwise – I would agree), ontology, and the semantic web, as well as best bets and user metadata. He next went over a framework. Level one has such things as key words, level two has such things as a thesaurus, and level three has facets. Level four is a knowledge map.

The people dimension includes how to deal with all the “tribes” in an organization. Each group has its own vocabularies and culture with values. They also have different models of knowledge. There are both individual people and communities to deal with. Also a central team needs to be put in place to support the semantic infrastructure. This team needs to be cross-organizational and interdisciplinary. There are various technologies that come into play such as text mining and search-based applications.

The business processes analysis needs to include knowledge architecture experts since process SMEs tend to be bad at knowledge structures. The knowledge map is the foundation for this work. There are various techniques to put together a knowledge map such as interviews. In Phase One you start with a high-level structure inventory and content structure, as well as a map of the organization itself. You need to identify the SMEs in each area and secure access to them.

Next, in Phase Two you spider the relevant content to explore it and categorize it. You work with SMEs to better understand the content and how it is used. In Phase Three you develop the knowledge map and refine it by exposing it to people. You also create an expertise map.

Tom said that enterprise taxonomies are a dead end. They are often expensive and difficult to apply.  Maintenance is also difficult. I would agree here as the world is constantly changing. Tom said that instead of giant taxonomies, you create small ones with intelligence built in so you keep up with changes. Let the tools such as text mining help you keep up. The key is to have a big picture of how things fit together but not a rigid structure. Then you can adapt wisely.

Tom next went over the benefits. First, a semantic infrastructure allows you to handle the massive amounts of unstructured content.  He said that this gives it some structure to make the content useful. Google cannot just do it. Google works well on the Web because of its link structure that adds a human element. However, enterprise is not linked in the same way as the Web.  Google also has thousands of editors and it actually uses taxonomies but does not publicize this. Within the enterprise Google does not bring its Web tools.

Semantic technology makes the enterprise task possible. The benefits can be huge. They can be so large that people do not believe them. People waste too much time looking for information or re-creating information that was already there.  The cost of missing the right content can be even larger. There are figures that say it cost about $12 million per thousand employees wasting time looking for stuff.  Recreating existing content can be $4.5 million per 1,000 employees.

Another benefit is getting better return on technology investments as most enterprise content management under-perform without a semantic infrastructure. For example, authors are usually reluctant to tag their material and are often poor at it. Text analytic tools can suggest tags and then authors can simply approve. This works better than having the authors start from a blank slate. This approach can be better than going out to buy new content management tools and still having the same problem. Tom said that enterprise search is static but building search-based applications is expanding. 

Tom said that companies organize their people. This is not questioned. The same logic indicates that they should organize their content. This is a nice point.  At the same time, I like his point that a top down rigid taxonomy does not work for organizing content.  This is also consistent with the new ways of managing people. 

Tom next covreed text analytic tools that make the semantic analytics possible. He also covered how to evaluate text analytics tools.  These tools begin with noun phrase extraction and this is used to build categories. It also feeds facet development. In addition, they can do summarization of documents.  There is also fact extraction. A new hot area is sentiment analysis.  

Auto-categorization is the core of these tools. One issue here is disambiguation.  You usually need to train the tool and continue to refine it. A set of rules is built to help determine meaning.  Tom showed an example for looking at call center rep notes.  There were rules on such things as proximity of word combinations to help determine meaning.  You can go and see how documents are categorized and make refinements. Sample rule: if the word begins with a capital letter and the next word is “says” or “said” it is likely a person.  There may be exceptions to specify.

With sentiment analysis you need to go beyond positive and negative, as there are levels of these dimensions.  Most sentiment analysis involves finding positive and negative words. With categorization you can build more complex rules and analyses.

Tom next covered how to evaluate text analytic tools. You need to start with the business context, like any tools. Then you look at existing tools to see what is available and what the new tools need to interact with it. Text analytics is different that most software so the IT department can be challenged to handle this evaluation task. The business players may know the content but need help on how to use the tools.  The team needs to be interdisciplinary.

Two-phased approach: Start like traditional software evaluation: what does the industry say about tools, do a feature scorecard and have a short list for demos. Then do a proof of concept.  These proof of concept efforts usually take 4 to 6 weeks. You need to go beyond a vendor’s demo to see what happens with your content. In your proof of concept evaluation, you need to see various refinement cycles as the tools need to be trained and will not work out of the box. You need to see if the auto-categorization works with your content. With entity extraction you need to see about scale and disambiguation.  When you look for vendors you can find standalone text analytics tools and those embedded in content management and search tools. You also need to consider that categorization is iterative so you need to adapt over time and provide budget for it. There cannot be a single total score in your proof of concept comparisons but a matrix of things to score on.

No aspect of this is simple and it does not end, as it is infrastructure that continues to evolve. The complexity of semantic infrastructure creation opens up many risks on the people and cultural side, as well as the usual suspects on the technology side. Each area brings is own people baggage and there are many areas to involve. Tom gave an example of how this can go wrong. It is more about anthropology than library science because of the people issues.

I mentioned that an issue in complex implementations like this is that many people can say no and often no one can say yes. You need to have executive sponsorship lined up to deal with the naysayers.

Next we looked at applications starting with search.  The software should not start and end with search. You need text analytics and text mining. Rich search results are the result of conversations. You start the search and refine it and let people choose facets for this refinement. Facet navigation is advanced search for non-advanced searchers. It lets you get more sophisticated. Two areas for the combination of semantic technology and search are eCommerce and within the enterprise. The Web itself is too complex.

Sample use cases: finding duplicate documents, text mining, content aggregation, combining with data mining and others.  Getting more specific you can look at customer sentiment in call center logs but this can be challenging. Tom went through the development process for sentiment analysis. There is a training process here also and it can be complex. You start with general rules and then create more sophisticated ones to get more precise.

Future directions include greater integration between advanced tools such as text mining and predictive analytics. Another area is combining text analytics and text mining. Semantic infrastructure is the way to make effective use of these new tools.  Tom said Watson was just the start. It seems like there is a lot of promise in this space and a ways to go to fully realize this promise. 

 

 

The New Sweat Shop of the Mind: Content Creation

One of the most significant intended consequences of the success of Google’s Page Rank is SEO and in its darker form, black hat spam. I reported on a milder version of this in Trying to Game Google on Mother’s Day. The New York Times covered a darker side with its article, Google’s War on Nonsense. It reported on a new form of sweat shops, the content farms. As they wrote: “Content farms, which have flourished on the Web in the past 18 months, are massive news sites that use headlines, keywords and other tricks to lure Web-users into looking at ads. These sites confound and embarrass Google by gaming its ranking system.”

The Economist described two content farms, Associated Content and Demand Media, as cleverly cynical operations that “aim to produce content at a price so low that even meager advertising revenue can support it.” The Times goes on the write: “The insultingly vacuous and frankly bizarre prose of the content farms — it seems ripped from Wikipedia and translated from the Romanian — cheapens all online information.”

 It sounds like machines create such prose but actually, as the Times reports,  “they are written by writers who work like robots. As recent accounts of life in these words-are-money mills make clear, some content-farm writers have deadlines as frequently as every 25 minutes. Others are expected to turn around reported pieces, containing interviews with several experts, in an hour.”

 Why does this occur? Because Google uses algorithms that attempt to impose order on the chaotic universe of the Web. As Nick Carr quotes Google CEO, Eric Schmidt, in his Atlantic article, Is Google Making Us Stupid?, that it is “a company that’s founded around the science of measurement,” and it is striving to “systematize everything” it does. Carr adds that what Fred Taylor did for the work of the hand, Google is doing for the work of the mind. Nick goes on to write, “in Google’s world, the world we enter when we go online, there’s little place for the fuzziness of contemplation. Ambiguity is not an opening for insight but a bug to be fixed.”

 In response to this the Times writes that these writers attempt to work within Google’s quest and, “cram together words that someone’s research had suggested might be in demand on Google, position these strings as titles and headlines, embellish them with other inoffensive words and make the whole confection vaguely resemble an article.” All of this is done in the interest of clarity and avoiding ambiguity?

 Instead of sewing cheap clothes or building poorly made toys in bad conditions these workers are attempting to build content in bad conditions which is the outcome of doing, as Carr wrote, “what Fred Taylor did for the work of the hand, Google is doing for the work of the mind.”

 These writers are not interested in journalist quality but, as documented in The AOL Way, “a company document leaked to the press not long ago. That document reduces the art of journalism to a process that begins with using metrics to “identify high-demand topics” and ends with the review of this “hi-vol, lo-cost” textual content — those are articles, folks — for such important literary virtues as Google rank and social-media traction.”

To its credit Google is striking back. As the Times reported, “the corporation concocted what it concocts best: an algorithm” to sift out the content farm content. And the SEO war continues. So far Google is making progress but they are up against a highly motivated set of foes. The war will continue as long as some version of reputation, relevance, and recency remain in play to determine what you get to see first.  

To be fair Google these sweat shops of the mind an unintended consequence and I am sure Fred Taylor would not like sweat shops of the hand. This is also not really an attack on Google. I used it every day and it is my home page. I used it to write this article. It is the best source I know when I know want I am looking for like when the Red Sox game starts or the name of the Thai restaurant in my hometown.  I do not know of an alternative that works better but we should be aware that measurement has its place and cannot provide the perfect answer. That is an illusion that most statisticians will agree with.

Google’s algorithms also have their cost. While most serious online writers do not act like the content farms, there is a small portion of their cognitive space that does keep SEO in mind so people will find their articles. I do it. This activity brings no real value outside SEO and keeps a portion of our brain working in the new sweat shop of the mind. 

When the phonetic alphabet was created it freed up portions of our mind form having to remember so much content. What will be the next breakthrough that frees us up from using a part of our cognitive space for SEO?

 

Forrester on Enterprise Search Trends

Here is a new report on 2011 enterprise search trends from Forrester, Enterprise Search: Six Key Trends to Watch by Leslie Owens with Stephen Powers and Anjali Yakkundi. The report indicates that despite the fact that only 10% of IT leaders will upgrade or expand their information access implementations this year, search experts are optimistic about their ability to deliver search solutions that are both usable and useful.

What is the source of optimism?  Search technology is now well established, and the standards set by Consumer Web search  simplifies design decision and training. As Leslie and her colleagues write, “First-generation enterprise search was not easy to use and produced unsatisfactory results from a usability and relevance standpoint. Today’s knowledge workers demand role-specific, contextual search everywhere they work.”

As the industry standards for search evolve, the report predicts that vendors will change their products to adapt to new customer investment trends with changes in semantics capabilities and increased usage of search-based applications (SBA). Cloud and mobile are certain to have an impact on search programs as well, but plans for standalone enterprise search services in the cloud lag far behind cloud-based email or collaboration.

Here are the six trends:

Search managers will initiate business conversations, not gather requirements. This is a position that all parts of IT should take.

IT will apply search to reveal aggregate workplace patterns. It will be become increasingly recognized that the knowledge and insight revealed on social platforms and enterprise 2.0 via technologies like idea management and micro-blogging tools is strategically important. So access to this great content will be necessary.

Business leaders will dictate the scope of search. Business should have a role in all IT issues as search becomes a more pervasive part f enterprise technology.

Quality content will be the focus, not additional metadata. You want to search technology to narrow results as content explodes.

Knowledge workers will demand transparency and democratization of the search experience. This is part of the general enterprise 2.0 movement.

The enterprise search UI will no longer resemble the consumer Web. Enterprise search will be embedded in business processes rather than in a standalone white box. This is an excellent move. Embedding other functionality such as activity streams in work processes will also help productivity.

There is more on each trend and you can find the report on their web site. 

 

Our Obsession with Scoring is Now Taking on Individuals on the Web

I had not been paying too much attention to Klout until I saw this article in the New York Times, Got Twitter? You’ve Been Scored. I skimmed it and tweeted about it to save the link for further reading. I often use my Twitter feed, @billlives, as a social bookmarking tool so I remember and have access to posts and articles I want to save. I then go through my tweets twice a month and it gives me an overview of what I thought was important during this time. Then I do a blog post that lists the tweets I found really important as a way archive these tweets. I am sure there are more efficient ways to archive Twitter but I find this practice a useful exercise. I also use the @darwineco Twitter feed for the same reason when the topic is more relavent to Darwin Ecosystems with the added goal of sharing it with my teammates at Darwin.  Then I follow the same steps ending up with a post on the Darwin blog.

Well, the tweet on Got Twitter? You’ve Been Scored got retweeted six times within a few hours so this action caused me to look more closely at the article. I then tweeted that I was inspired by the RTs to write a blog post on the article and I got two replies for the six who said they were looking forward to the post. I was pleased as this type of exchange is how Twitter is supposed to work. I also met a few new Twitter friends in the process.

The article itself exposed me more to the growing movement behind such tools as Klout and Peerindex. It is about all of us getting a number, similar to Robert Parker’s quantification of wine. While there is more objectivity here that with Parker, the underlying motive is the same, the quantification of the world. This is a topic I have covered several times.  I did a four part series on this blog that began with Rising Above the Over Quantification of Content: Part One: Parker vs. Piaget.

In this post I quoted, Adam Gopnik in the September 6, 2004 issue of the New Yorker, “(Parker) was uncannily successful because (he was an) apostle of a radical American empiricism – an insistence that facts and numbers could show you what was really going on, against everything tradition told you…The debate is not about whether the numbers are right but whether it is right to have numbers.”

In a similar perspective, Nick Carr quotes Google CEO, Eric Schmidt, in his Atlantic article,Is Google Making Us Stupid?, that it is “a company that’s founded around the science of measurement,” and it is striving to “systematize everything” it does. Carr adds that what Fred Taylor did for the work of the hand, Google is doing for the work of the mind. I would add: and what Robert Parker did for wine.  Nick goes on to write, “in Google’s world, the world we enter when we go online, there’s little place for the fuzziness of contemplation. Ambiguity is not an opening for insight but a bug to be fixed.”

Personally, I think a little ambiguity and complexity is good. So how does this relate to someone’s reputation on the Web? Personally, I follow people for many different reasons so a single score is not going to help too much. I also follow people related to three different twitter accounts I am connected with (@billives, @darwineco, and @outstart). The fact that Klout does offer the top ten topics a person is known for helps but there is still a gap between the complexity of why I follow people and a score, in the same way I might like a wine for many different reasons, occasions, and food parings.

The New York Times article quotes, Jeremiah Owyang, an analyst I respect with Altimeter Group. “Using a single metric to evaluate influence is dangerous. He noted that Klout “lacks sentiment analysis” — The Times adds, “so a user who generates a lot of digital chatter might receive a high score even though what’s being said about the user is negative. Also, a single metric can be misleading: someone with little Twitter experience can snag a high score if they happen to post a video that goes viral.”

I look at the Klout scores of people I know and there is a rough correlation with my views of what works for me but certainly not a precise one. Like wine, I think this is an area best left to the full range of human cognitive abilities and the multiple associations the human mind can make.  The Klout score of someone I do not known might help except for the fact that I know it is highly likely not completely accurate for my needs, only a clue about what to pursue in more depth. This also does not consider the opportunity to game this score, just as people do with Google. For the record, I did check my scores on two of these tools and my Klout score is 55 and my Peerindex score is 59 but I am not really sure what this means and I do not know how to game it. 

At Darwin, we take the opposite view of Google and Klout. We view the Web as too massive to understand and human reasoning too complex and individualized to fully progam. Rather that drawing inspiration for Fred Taylor’s time and motion studies and attempting to understand and control the universe of content on the Web, Darwin is inspired by Edward Norton Lorenz’s work on Chaos Theory in mathematics and weather prediction. Darwin does not try to impose an order on content, it lets content self-organize. During the 1950s, Lorenz became skeptical of the appropriateness of the linear statistical models in meteorology, as most atmospheric phenomena involved in weather forecasting are non-linear. Instead he proposed the concept of attractors that operate within a dynamic system that evolves over time in a complex, non-repeating pattern.  This is the complete opposite of Fred Taylor’s repeatable time and motion approach.

Now I am not saying the Darwin approach is better than the Google or Klout approach, just different. We actually think the two approaches complement each other. We call our tool an awareness engine™ to distinguish it from a search engine. The Darwin approach does not require human intervention to be organize the content; it is self-organizing based on the content itself and therefore more representative of what is happening as it occurs, and respectful of each unique context. It also eliminates the possibilities of search optimization and spam. It visualizes the content in a way that allows you to sort through the complexity a but better and make your own decisions on what is most important and relevant to you. 

 

 

 

Is the Influence of Search Engine Rankings Over Our Content Access a Good Thing?

Dick Singer recently wrote the post, Pay Attention to Search Engine Rankings: Top 3 Spots on Google Get 58% of Clicks, on Social Media Today.  He notes that, “Optify (via Searchengine watch) carried out a study, which showed that websites at the top of Google get an average of 36.4% of clicks, those in second place 12.5% and those in third 9.5%.” and adds, “That chimes in with an earlier Chitika study, showing that the no 1 spot on Google gets 34.35% of traffic.   Optify also says that being any further back than page two has no real business value for you at all.” The write works for a social media agency and beleives in SEO.

I am not sure this is a good situation. Now I have to admit that at Darwin Ecosystem we also play the SEO game also. It is a necessity if you want to get your content noticed. If you do a Google search on the term – awareness engine – you will find the Darwin Awareness Engine™ generally in the top six spots. I also turn to Google to find things when I know what I am looking for such as the latest Red Sox game score.

However, this situation means that those who can afford an agency to help them with SEO or possess that skill themselves will get their content on top.  There is a skill between people and content that is not related to the content. This was the situation in the middle ages when scribes controlled the production of content. The printing press opened things up but of course you still needed a printing press or the money to hire one. The Web was supposed to further open things up with user-generated content. To a certain extent it has but you still need to find things and search engine algorithms are the new scribe’s pen that stands between you and the ability to get ideas noticed.

With the Darwin Awareness Engine™ we try to go beyond search engine algorithms to let the content self-organize by using algorithms based on Chaos Theory rather than variations on popularity.  When I want to find useful articles and breaking news in a specific topic, I now turn to the Darwin Awareness Engine™ rather than Google. I use it to find things for my Twitter feed and for this blog and other blogs I write for. That is how I found Dick Singer’s post. I doubt if I would have found it through Google, at least not so easily. 

 

Attempted Gaming of Google on Mother’s Day

I hope you had a good Father's day. Last month the New York Times ran the article, Trying to Game Google on ‘Mother’s Day Flowers, that demonstrates the unintended consequences of Google’s approach to search. Of course, Google remains the gold standard of Web search but this is in part, because of its constant watch for cheaters. The article reported that internet marketing experts say that several well known flower sellers are trying to elevate their Web sites in search results with a strategy that violates Google’s guidelines.

While the companies deny it, they all have links on Web sites that are riddled with paid links, many of which include phrases like “mothers day flowers,” “mothers day arrangements” and “cheap mothers day flowers.” As the Times said, anyone who clicks on those backlinks, as they are known, gets sent to the floral retailer who paid for them.

This is risky because if Google catches companies in link-buying schemes, they are often penalized by being sent way down in its search results, sometimes for months. When confronted by the Times, Google’s Jake Hubert, sent this statement: “None of the links shared by The New York Times had a significant impact on our rankings, due to automated systems we have in place to assess the relevance of links. As always, we investigate spam reports and take corrective action where appropriate.”

In other words, while they were trying to cheat, Google felt they were not effective so they were not going to penalize them.  This is like saying a badly botched robbery is not a crime because of the incompetence of the crooks. But then this is not an actual crime, just a misuse of links to game a system that is susceptible   to it. 

The Times quoted one person on conditions of confidentiality that she received $30 a month to post a “mothers day flowers” link on her home page. Of course, as the Times notes, “it is impossible to double-check Google’s conclusion that few of the links of the florist companies helped in search results. The particulars of Google’s algorithm are shrouded in secrecy for the same reason that a bank does not publicize the route to its vault.”

This is why we use a different approach for the Darwin Awareness Engine™.  Rather than using an external framework that is open to gaming to organize search results, The Awareness Engine makes use of Chaos Theory based algorithms that let the content self organize. You cannot SEO or spam it unless you control the entire set of targeted content.  Links to do not matter, only content. Now, the Darwin Awareness Engine is not designed to replace Google but it can be used to complement it and find the content that you might not expect or know to look for in an area of interest. 

 

My Notes from the 2011 MIT Sloan CIO Symposium: Enterprise Analytics = Business Values

Here are more of my notes from the MIT Sloan CIO Symposium: Enterprise Analytics = Business Values. Participants included: Moderator: Michael S. Hopkins, Editor-in-chief, MIT Sloan Management Review, panel: Rock Gnatovich,
SVP & COO, Spotfire, TIBCO Software Inc., Brad Peterson, CIO, Charles Schwab, Sid Probstein, CTO, Attivio, and Renée Romano Nocker, Director of Technology Product Marketing, SAS.

Michael said that in 2009 more digital information was produced than in all of past history – so much to look at. They are doing a survey on uses of data at the Sloan Management Review. He asked a survey question – imagine an organization transformed to effectively use data. How close to this ideal is your organization on a scale of 1 – 10? Also where is management in general on this? He asked the panel first. 

Rock answered in terms of level of satisfaction of some of their customers.  He said that business users are finding traditional BI tools too hard to use. However, some of their customers are using their tool, Spotfire, real time and would rate it 8.5.

Brad said that at eBay, his prior employer, all the IT solutions did not work at first because of the scale but it has been solved and now many of the employees have left to bring this knowledge elsewhere. He would give eBay an 8. Data mining was big for them but it was new at their scale. At Charles Schwab he would say a 6. In financial services you need to be very precise with customer data. They want to go to social and mobile next. What they need to do is drive marketing growth more.

Sid said that at Attivio he would say they are at a 7. They are a young company and not globally dispersed so some data collection is not hard. Michael asked about the business world at large and Sid said it is likely at a 5. It is hard to get data into the funnel. Sid said that the interest in unstructured data has grown in the past two years. Renee said that since coming to SAS she is seeing more of what is possible so she world rate the overall business world at a 3 or 4.

Michael said in their over 3000 responses to the survey so far the average is about 4.5. These numbers are likely high because of a company pride issue. Thought leaders would say a 1 or 1.2 and there is great room for growth.  When asked what are the obstacles the answer is not data. The big obstacle lack of understanding how to use the data to solve problems and embed data into business processes.

Michael asked about the biggest misconception about using data. Renee said when business leaders hear analytics there is a focus on reporting and measurement. She sees the value in predicting, not looking back. She sees there is a lack of education in the value proposition on using prediction to make changes in anticipation.

Michael asked if they have seen improvements in the ability to make changes based on data. Sid said there are now offices to manage the data so there is ownership. Analytics are about optimization and changing pain points, not looking backward.  Companies need to do more experimenting.

Michael asked Brad about what obstacles he has seen as he is the one non-vendor on the panel. Brad said that much of senior management is not data oriented. He was trained in the value of data at MIT. We need more senior management to get it.

Rock said there is a sense that only a few people can do analytics in an organization. The IT organization has not been responding fast enough so people bypass it. His firm’s client base has a lot of people who do not trust the data they are given and want to work on data itself.

Brad said that at eBay they had to be able to sell anything ever built. So they could not put in place highly structured data gathering. They had to adapt. You have to give up on too much organization or the business will not adapt. Renee agreed that being open is good in many cases but you still need structure in others. IT people get excited when they are using innovative analytics to help drive decisions in their company.

There was a question from audience about being creative and not getting a mess. What is the right balance of empowerment without chaos. Rock said Excel, for example, allows you to do anything you want but you cannot share it as no one else will know what you did.  You need a combination of guided workflow and free form.

Another question: Some people need to simply have the forms filled in and others are hungry for more data. What is the thing to move companies from a 4 to an 8 on the scale. Renee said it is the skills on how to pick the important parts of data to look at. It is the analytic skills more than the technical skill. Sid agreed that expertise is an issue but also infrastructure requirements and the corporate need for short-term results that limits investment. Companies do not want to make the investment. Brad said the you need to get funds from marketing budgets as this is where the value will go. Rock said it is a combination of technology and methods. He agrees it is a skill gap in the analytic space. Also, the need to not have to organize everything opens up for innovation.

Audience: I heard this morning that BI is not widely adapted and is still hard to use.  How to get this adaption.  Sid said you need to get wider use for BI tools. You need to shift from complex reporting to Google style simplicity. We would applaud this at Darwin Ecosystem as that is one of our design goals. Sid added that you need the consumer model of simplicity to increase usage

An audience member said he has to sell BI within the organization. He asked for any practical ideas to sell senior management.  Rock said you have to create a virtual environment where people can bring their data.  You then need to allow for socialization as people trust other people more than data.  Renee said you need good communicators. They went out and found problems that need data to solve them. Then you do pilots on these to demonstrate value. Brad said give business people an iPad with the BI tool on it and they will love it and show it to others.

 

 

The Evolution of Site Search

Site search is becoming more than a means to increase website traffic. Now 40% of content and collaboration professionals see the feature primarily as a tool to improve customer self-service, according to a new Forrester report, Site Search Evolves from Technical Feature to Customer Touchpoint, by Leslie Owens with Stephen Powers and Anjali Yakkundi.  The next objectives were: increased sales conversion 30%, increased search traffic (SEO) 12% and increased page views 10%.

However, the report also finds that most corporate websites provide an inadequate search experience, lacking customer friendly self-service in areas such as speed, adaptability, relevance, content integration and task fit. In fact, 69% of Web sites examined did not even offer searchable self-help for customer service. Asked to rate the overall success of their search initiative, a survey of 188 content and collaboration professionals gave an average score of just 6 out of 10, reflecting a need for better, more measurable value from search. 

The report found that two thirds of search decision-makers are planning to expand their implementation and usage of site search this year. To be successful, according to Owens, IT and marketing teams must collaborate to develop the top three improvement areas in search: better relevance, enhanced usability, and quality content. They need to move the search functionality from a siloed activity to a cross-functional search partnership.  I would add that this business involvement is the success formula for most IT initiatives.

The report suggests that as marketers begin to take a greater role in search vendor selection and administration in the coming years, they may even completely take over tasks like boosting and blocking content, and setting up metrics to gauge growing levels of customer experience.

Forrester sees site search as a key area for investment. My sense of the market and my own experiences with Web sites has be agreeing with them.  Web sites have often replaced people as the face of the enterprise. It makes sense to put your best face forward. I was pleased to receive a review copy of the report. You can see the report at the Forrester site.

 

 

Forrester: Enterprise Search is Maturing and Becoming More Social

Yesterday I wrote about social search and the Web. Forrester has provided an interesting new report, Enterprise Search: Six Key Trends to Watch by Leslie Owens with Stephen Powers and Anjali Yakkundi, and social is more positive here. Enterprise search used to be difficult and often unproductive. Now knowledge workers expect role-specific, contextual search everywhere they work and this is becoming more possible. The report notes that search technology is mature and stable; for the most part, bottlenecks regarding scale, security, and connectivity can be easily resolved. I was pleased to receive a review copy.

Interviews with thoughts leaders and results from surveys indicate that despite the fact that only 10% of IT leaders will upgrade or expand their information access implementations this year, search experts are optimistic about their ability to deliver search solutions that are both usable and useful. The report attributes this optimism to the fact that search technology is now well established, and the pervasiveness of Web search simplifies design decision and training.

As the industry standards for search evolve, vendors will change their products to adapt to new customer investment trends with changes in semantics capabilities and increased usage of search-based applications (SBA). I have also seen a rise in SBAs. Search vendors are making it easier to develop search based applications on top of their platforms via prebuilt components and templates.

Cloud and mobile will have an impact on search programs. However, plans for standalone enterprise search services in the cloud lag far behind cloud-based email or collaboration.

Despite this progress, ROI remains difficult to determine and prove. Usage metrics do not readily translate into bottom line figures. However, costs are coming down as Web services reduce the cost and intricacy of crawling enterprise systems. With W XML connectors are becoming easy or even not necessary.

Social media, especially activity streams through enterprise micro-blogging, is on the rise. The authors feel that as enterprise social computing grows, the knowledge and insight revealed on social platforms via technologies like micro-blogging will become strategically important. I certainly agree and have seen this happening in a number of cases. Organizations will use search to provide access to and analysis of the real-time flow of internal social content in forums and threaded discussions. This is an important source of knowledge capital that needs to be made accessible.

Social aspects will also impact search as more social features come into practice. The authors predict that knowledge workers will also increase their power over enterprise search as they tag, recommend, rename, and re-rank search results on their intranet. Social behavior like tagging and recommending will also become a signal for relevance calculations.

Search and social are very interrelated. I think this combination will continue to be more integrated. The report offers more and I recommend taking a look. 

 

How Will Social Search Impact Web Search and Help Find Good Content?

Over ten years ago Google added a new and transformative dimension to search with its page rank concept. Let the readers help determine what content has the highest value and also look at currency, along with relevance   This has become very useful. I use it every day and it is the default home page for my laptop. However, the top results on any given search are often dominated by the sites using the best SEO techniques.  We have all experienced this. I also have used it to my advantage with some of the blogs I write and helped some of my consulting clients with this effort.  Of course, I only use legit techniques as do most web site people but not everyone does. I mainly try to provide good relevant content and let the readers respond. Google does its best to police any bad behavior and stay ahead of the spammers and black hat SEO perpetrators.

As discussed in: How Will Social Search Affect Social Media and SEO?, over the past few months, “Google and Bing have started to allude that they are beginning to increase the impact of social media content and links on their search results, citing that social networking activity is a strong indicator of what might be most relevant to consumer searches.”

This post looks at some of the impacts of this move. Among those cited were, increased importance of social media buzz around brands/businesses, increased importance of social media links, and more brand benefit from search engines. In other words social media, will be become an increased playing field for SEO. The word optimization was used at lot. The brands and sites that are best at SEO within social will prevail, not necessarily those with the best products or ideas.  This move may likely make it harder to us to find the really good content as it gets buried beneath the mounds of optimized content.

 

 

Using Semantic Technology and Machine Learning for Consumer Advice: Retrevo

With all the publicity over IBM’s Watson, there has been a lot of discussion about how far computers can go to “replace” people. Of course, I think that is not really the question to ask. It should rather be conceived as how computers can complement people. This complementary nature is also the perspective held by Dr. Aditya Vailaya, Chief Scientist for Retrevo.com. He has a PhD in Computer Science with a specialization in Machine Learning and Statistical Pattern Recognition, as well as significant experience with business applications of this technology.  We recently discussed the role of computers and people in handling complex topics that require some level of both learning and semantic analysis.

I have often covered Retrevo’s creative surveys and other work such as these posts: Electric Gadget Inter-City Challenges and Are You a Digital Spy? So I was very interested to learn some about what goes into their services.

Aditya pointed out that Watson has demonstrated that you can build a machine to handle some level of cognition. However, this takes considerable effort. Watson was built and trained by a team of experts over a number of years. It uses math algorithms. These coupled with semantic analysis allow it to understand a natural language question and determine the probability that its answer is correct. However, it is good for a very specific task. The years of training may make it better than most, if not all, humans in playing Jeopardy. However, it will fail against humans in most of the other tasks we face every day.

To put the human versus computer issues in some more perspective it is useful to look at the work of pair of researchers, Martin Hilbert (USC) and Prisicilla Lopez (Open University of Catalonia) who have been looking at the growth of computer power and storage over the past twenty plus years. Recent research noted that, "to put our findings in perspective, the 6.4*10¹⁸ instructions per second that human kind can carry out on its general-purpose computers in 2007 are in the same ballpark area as the maximum number of nerve impulses executed by one human brain per second.”  So all the computers in the world have now reached the capacity of one person. Congratulations. Adtiya feels that these numbers are in the right ballpark.

So the issue is not whether computers will outpace people but how the two can work together. Computers are very good at doing boring tedious, repetitive tasks than would drive people crazy at a rate and scale far beyond what people can do even with a fresh start on their best days. This frees people up to do the more complex and interesting tasks.

James Taylor makes a similar point in his post, Decision Management in the New York Times, commenting on a New York Times article, Smarter than you think, on e-discovery. He concludes that, “automation of decisions sometimes reduces the need for staff. Much more often it innovates and allows companies to apply the same staff to more problems by replacing boring, mechanical work with more interesting, more difficult work that is hard to automate or where automation is not desirable.”

Aditya noted that as computers are able to handle increasing complex tasks this frees people to do increasingly more interesting tasks. He also noted one of the main differences between people and computers. People have motivations. They have the will to survive, improve themselves, and many other things. Despite 2001 Space Odyssey, computers do not have feelings or motivations.  Perhaps this lack of motivation is one reason they do not complain about long work hours but even that idea might be giving them too much intelligence.

Aditya said that Watson takes what the e-discovery tools do a step further. They just run search queries and use semantic analysis to sort what is relevant. Watson takes in full sentences in a natural language format and provides results back in a similar format. However, it can still make errors, just as the e-discovery tools often deal with the probability of being correct.

As these tools require extensive training to get started and constant re-training to stay on target there remains a strong role for people even within the use of these tools. Perhaps most importantly, people need to determine the questions these tools are targeted to address.

I asked Aditya how Retrevo uses machine learning. He said they provide information to consumers on products available on the Web such as digital cameras. To offer the best information, they use computers to scan the Web for product information and reviews. Addressing the volume of content out there is certainly more than a person can do or even a large team of people, especially at the speed required to stay current.

This is a great example of the right place to use machine learning and semantic technology.  People are still required to train the computers, point them at the right sites, and monitor the results to determine adjustments to the algorithms.  I would trust Retrevo’s computers’ ability to find the wide array of current options and viewpoints in product category over a team of people.  It is then up to the people at Retrevo to keep pointing them in the right direction, as well as pushing the capabilities of the computers further.

 

 

Disaster in Making: Letting Google Influence on How We Cook

Here is a horrible sign of the dumbing down of the world and that also exposes a flaw in Google’s popularity ranking approach. TechCrunch recently covered on how Google’s Robotic Recipe Search Favors SEO Over Good Food. It reported,” “every month about a billion of its searches are for recipes. The dishes that its search engine turns up, particularly those on the first page of results, have a huge impact on what Americans cook.”

Unfortunately a perhaps well-intended move by Google could make this a food disaster. It announced that it was adding a new kind of search, specifically for recipes. It inadvertently sided with the quick-and-easy faction against the cooking-matters group. As tech Crunch reported, “The problem is that this new search effectively prevents the thousands of excellent cooking sites and blogs from ever seeing the light of day. More importantly, those smaller sites and blogs are where much of the best work in food is happening online.”

I have written about food blogs a good bit in the past and find them wonderful. Here are a few sample posts, Exploring Food Blogs, More on Food Blogs, and List of Food Blogs. One of my favorites is Russelnod: the way we ate then to the way we eat now, written by my good friend Don Lesser.

TechCrunch gave a great example of the consequences of Google recent move. Doing a search for “cassoulet,” the French dish with beans, meat and other good things, you can refine the results by time. However, your choices are “less than 15 min,” “less than 30 min,” or “less than 60 min.” There is no option for more than 60 minutes. A classic cassoulet takes at least 4 hours to make. What ever happen to Slow Food?

The TechCrunch post concludes. “If they don’t change their current approach, I fear to contemplate the future of American cooking.” Me too.

 

Overcoming Information Overload on the Web: Part 3: Organic Semantics

This is the third of a three part series on how the Darwin Awareness Engine™ addresses some of the key problems generated by information overload on the Web. The Awareness Engine is a Web aggregation utility for anyone needing to maintain awareness of emerging events from the Internet and intranet.  The technology provides a temporal semantic visualization of events, by means of a Scan Cloud™ that facilitates a rapid understanding of trends, outliers and anomalies.

Issue Three: Organic semantics:

The Semantic Web and semantic analysis require human classification intervention and semantic model algorithms to infer and interpret meaning. Emerging concepts such as folksonomy and new assumptions that drive analytics challenge these semantic technologies.  The challenge is rooted in the detection of new concepts, and the application of evolving analytic approaches for new business usage.  The current model is robust but rarely resilient because it relies on machine-driven inference of meaning. 

Our technology’s use of temporal semantic correlation reveals the evolution of correlated patterns of terms.  It does not claim to know the meaning of the information, but simply displays what is.  This allows for the meaning to be inferred by the user.  A by-product of our technology is an organic semantic model that can be used to see the relationships that exist between terms over a period of time.

This representation accelerates the identification of terms to be included in Semantic Web classes, but also captures the intensity of relationships over time in an organic semantic web.  The use of such a resource can help in creating recurring pattern forecasts and improve variation detection for semantic analysis tools.  

This concludes our three part series on combating information overload on the Web. We feel that our key differentiator is our ability to overcome information overload by accelerating information consumption through synthesized visualizations outlining relationships and acceleration of emerging themes without attempting to analyze the meaning of the aggregated information. Although different in their delivery, intent and scope, our competitors are search engines, web aggregators and web analytic solutions.  

Our technology is being used to monitor and to be aware of emerging themes from: different industries, clients, political topics of interests, risk mitigation, business intelligence, brand awareness, journalist themes, enterprise 2.0 collaboration, conference buzz, twitter, etc.. It can also be used to improve advanced analytics and drive term-less search of the enterprise’s knowledge assets from our Scan Cloud navigation. Here is listing of Awareness Engine FAQs. 

 

Overcoming Information Overload on the Web: Part 2: Popularity Versus Temporal Correlation Ranking

This is the second of a three part series on how the Darwin Awareness Engine™ addresses some of the key problems generated by information overload on the Web. The Awareness Engine is a Web aggregation utility for anyone needing to maintain awareness of emerging events from the Internet and intranet.  The technology provides a temporal semantic visualization of events, by means of a Scan Cloud™ that facilitates a rapid understanding of trends, outliers and anomalies.

Issue Two: Popularity versus temporal correlation ranking:

Search engines operate on popularly ranking to deliver information.  This method is rooted in the Web 1.0 where information existed in great quantity and had a longer shelf life.  The Web 2.0 explosion generates information that is often isolated and relevant for a short period of time. This type of information may only appear in the 100th-plus page of search engine results.  In contrast, our temporal correlation-ranking model seeks the latest published information regardless of its popularity. This new model considers that the themes associated with the information are noteworthy if other related information is emerging across a defined timeframe.  This suggests an emerging or accelerating movement.  Darwin Awareness Engine reveals this emergence through its BuzzTape and ScanCloud.

Example:  The BuzzTape on Darwin Awareness Engine’s General News Edition displayed the relative acceleration of the term “Lou Dobbs”. Clicking on “Lou Dobbs” displayed a ScanCloud created from over 200 informal Web 2.0 social events showing terms such as “racist”, ”Latino Americans”, “petition”, “fire”, “CNN”  all created within the last 2 hours.  We searched Google News for the following five days without any coverage of this social reaction against Lou Dobbs. 

While being specific in our search, by adding terms that our technology correlated, we discovered related Web 2.0 documents buried deeply into the search results.  It took one week after noticing the social event for the press to address the issue and make the announcement of Lou Dobbs’ resignation over his indiscretions towards the Latino community.  Note that, in this example, we never programmed the system to specifically target Lou Dobbs.  The emergence was detected by our technology, as a result of his name being a term with great acceleration over just a few hours.

In the next post we will look at organic semantics

 

Overcoming Information Overload on the Web: Part 1: Too many sources, too many events.

This is the first of a three part series on how the Darwin Awareness Engine™ addresses some of the key problems generated by information overload on the Web. The Awareness Engine is a Web aggregation utility for anyone needing to maintain awareness of emerging events from the Internet and intranet.  The technology provides a temporal semantic visualization of events, by means of a Scan Cloud™ that facilitates a rapid understanding of trends, outliers and anomalies.

Issue One: Too many sources, too many events.   

As an information aggregator we target theme-specific sources through search and specific feeds.  Although accessible from on site edition, this body of information often remains overwhelming.  In contrast with traditional search engines, we do not first deliver the events in lists. Instead, these are presented in a Scan Cloud, revealing both the relative importance of terms as well as their inter-relation. This interactive representation makes it easier to understand the content without having to read through the content of the articles to understand “what is going on” and detecting emerging themes.  The user’s natural ability to infer meaning is leveraged by the tool.

Example:  During a presentation of Darwin Awareness Engine to a large energy holding company, the information curator, responsible for providing the CEO with reports on all public, political, and news events related to the company’s business and its owner’s public image, spotted the term “London” in the ScanCloud.  Most other terms and their highlighted relationships where within the expected context, but “London” was an anomaly (only someone with an intimate understanding of the company’s activities would have made this assessment). 

She asked to be excused for 5 minutes and returned to the meeting.  She shared with us that despite the tools currently available to her to comb the Web and the press, she had not noticed the event she just discovered using the Awareness Engine.  In fact, the event related to the violation of a UK court gag order that could have caused material damage to the company.  She identified the blogger and reported it to the Legal department. Subsequently she added the event to the CEO’s daily report.

In the next post I will look at popularity versus temporal correlation ranking as a means to uncover relevant content.

 

 

Recommind Supports End-to-End eDiscovery

Recommind provides predictive information management software through search-powered business applications. I last spoke with them in 2009 (see: Recommind’s MindServer Search™ 6.0 Addresses need for More Effective eDiscovery). Recently, I spoke with Craig Carpenter, Recommind’s VP of marketing and General Counsel. I noted the unusual job combination, and Craig said when your main clients are law firms and law departments within corporations, the combination helps. We mainly focused on their Axcelerate™ Review & Analysis suite with Predictive Coding™ and how it impacts eDiscovery.

First I asked Craig to define eDiscovery. It is a term that I have seen used both narrowly and broadly. Craig said it was initially used in a more narrow sense to refer to the process of preserving, collecting, reviewing, analyzing and producing relevant documents after a legal process has begun. Now, it generally extends to include information management in anticipation of possible legal issues, as well as during them, while also addressing regulatory needs. This reflects a more proactive approach that is being adopted by many Recommind customers. This broader approach means that content collection is not an isolated activity but one that is integrated into an enterprise’s total information management system on an ongoing basis.

Cisco is using Axcelerate to take a more proactive, strategic approach to eDiscovery. One of the drivers of increased eDiscovery costs is the explosion of enterprise content, which ironically Cisco is helping facilitate with its technology that simplifies the creation and transfer of large files, such as videos. Cisco wanted to have its own enormous stores of enterprise data easily accessible ahead of any legal issues that might arise. They also wanted greater cost containment and predictability so that legal costs do not dictate when they settle and when they fight a legal challenge.

Axcelerate offers customers like Cisco a powerful, end-to-end platform for eDiscovery management. First, Axcelerate’s Predictive Analytics™ capability automatically categorizes documents and enables deep analysis of the data at the start of a legal process, before major costs have been incurred. This greatly reduces the amount of expensive manual labor by legal experts as they decide which documents are relevant and/or privileged in a case. It can create a taxonomy on the fly based on the content and/or work with an existing taxonomy. Here is a sample Review and Analysis screen.

Here is a diagram of how the entire process works.

Second, Axcelerate’s Predictive Coding™ technology and workflow automatically analyzes, prioritizes and codes all documents in a collection. After a knowledgeable attorney reviews and codes a small number of documents, the software can automatically sort through all remaining documents on its own, in the process flagging all other pertinent documents while setting aside the rest. This dramatically expedites the document review process, slashing eDiscovery timelines and costs by 50%-90%. 

Third, Axcelerate’s Predictive Sampling™ methodology provides metrics on the accuracy of the Predictive Coding process, thereby making Predictive Coding not just the fastest review but also the most accurate and defensible. The availability of a specific accuracy rate makes courts far more comfortable with the process, as it goes well beyond what is considered “reasonable” – far more so than traditional linear review. Here is a sample Predictive Sampling screen.

Craig said Recommind offers both on-premise and private cloud options. The private cloud option has been growing fast (over 500% in 2010) as firms see the cost savings and flexibility of cloud solutions and become more comfortable with them.

Recommind offers other search-based applications as well. We briefly spoke about Decisiv Categorization, which many companies use for information management and governance. It uses Recommind’s patented Probabilistic Latent Semantic Analysis (PLSA) technology to look for patterns in the text within documents and automatically categorize them. I think that as the explosion of content continues there will be increasing applications for the technology that Recommind brings to the market. 

 

Looking at the Toyota Recall Through the Darwin Awareness Engine™ and Google Search and News

Here is an example of how the Darwin Awareness Engine™ can uncover breaking news stories and the shortcomings of traditional search for this task.  Now search is great when you know what you are looking for and I use Google search every day for this purpose. However, if you do not know what key words to put into your queries to find the unexpected a search engine falls short. This is one of the goals of an awareness engine.  Here is how it works using the Toyotas recall as an example. Thanks to our friend Romain Goday for this example.

The Buzz Tape™ runs across the top of the screen and shows topics of rising coverage in your target source, in this case general news sites and blogs.  Below we see that Toyota is a rising topic of interest.

Making Toyota the focus of a Scan Cloud™ brings up the term ‘recall’ as the predominate theme in the past hour.

Clicking on the term recall shows the related articles in the right column.

 Looking at one we see an article on the recall.

We can also explore other articles. In addition we can change to focus from “Toyota” to “recall” to see what else is happening in the recall space.  Let’s see what a Google search at the same time on Toyota brings.

There is nothing on the recall.  In addition searching on the term “recall” brings up nothing on Toyota.

In we combine Toyota and recall – “Toyota recall” – we find one article in Google news but we have had to anticipate this combination. The Darwin Awareness Engine does not require this anticipation and lets you discover the breaking story regardless of whether you were looking for it.

So we can see the traditional search engines have trouble with breaking news as it takes some time for items to become poplar and rise to the top. Google news helps but you have to anticipate trends to get good results. In contrast an awareness engine let’s you discover the unexpected. Rather than looking at a lot of RSS feeds, it aggregates the content scanning processes to single screen.  Here is how it helped the St. Louis PBS affiliate track in real time the conversations in the US immigration debate. 

 

Further Reflections on Dave Snowden’s Resilient Organization

Last November I attended the KM World 2010 the keynote session, The Resilient Organization led by Dave Snowden, Founder & CTO - Cognitive Edge. Here are my session notes. There were a number of things that Dave said that strike at some of the principles behind the Darwin Awareness Engine™. I wanted to reflect on those touch points in this post as we enter the new year.

First - Dave noted that most people can produce a decent meal through recipes and the right ingredients. A master chef can produce a great meal with whatever is your kitchen.  We need theory and not just recipes to blindly follow.

We developed the Darwin interface to be very simple to use. It only takes a few minutes to understand how to make use of the data visualizations. However, it works best under the hands of a master who really knows the content, can select the right feeds and make hypotheses to test through following feeds, Rob Paterson’s work with KETC is a great example (see KETC TV Social Media Experience Part Four: How Does Darwin Help Clarify Complex Issues?)

Second – Dave noted that the human brain scans a small percentage of the available data and makes quick decisions. The only people who look at all the data are autistic and they are overwhelmed but management theory adopts this autistic approach.  To react to the charging lion you cannot examine all the data before you react.

We designed both the Scan Cloud™ and the Buzz Tape™ for quick scanning (see below) only you to check out a vast amount of data quickly and rill down where you find interesting things. Anomalies show up.

Third - Dave said that despite what Chomsky said, there are no deep structures in language and so taxonomies are useful but limited.  We would certainly agree at Darwin since we let the content self-organize dynamically rather than impose an external structure. We are the opposite of semantic technology. If you let the content self organize there is no external framework like Google Page Rank to game or approach through legitimate methods. There is no SEO or even spam as it would serve no value. We allow the visualization of connections between content related to a particular theme to reveal new relationships and help with the discovery and awareness of trends, both anticipated and unanticipated.

Rather than taking a linear, deterministic approach to finding content through seeking repeatable patterns on the Web with an external framework like page rank as done by Google, we use Correlation Rank™ to find non-repeatable patterns in Web content that come and go in real time. Rather than using an external framework, we look for self-organizing patterns within the content itself.

Fourth - Dave noted that technology needs to be seen as cognitive augmentation.  Larry Prusack said intuition is just compressed experience. The human brain blends fragments in the face of challenge. He noted how General Patton did this in World War Two. We use provide the bits of content so the human mind can decide on how to put them together. Darwin Correlation Rank™ does not offer a precise answer, but it reveals a movement or trend as it occurs

Fifth - Dave stated that we need to move from aggregation for people to allowing people to synthesize fragments. We need to allow for contextual blending of fragments. We need to put decision makers in direct contact with the raw data. This is our goal at Darwin. 

You can download a podcast and slides of Dave's KM World talk at the Cognitive Edge site.  

 

Results for Using Search Queries for Future Predictions are Mixed

The MIT Technology Review recently reported on Future Gazing with Search Data in an interesting article by Katharine Gammon. She notes that computer scientists have praised Web-search data as a way to spot emerging trends. However, recent research shows that the results are mixed and context dependent.

For example, Erik Brynjolfsson, director of the MIT Center for Digital Business, says that Web search queries provide the most accurate predictions in cases where people do research before they make a purchase. His research has shown that a rise in home sales can be predicted from Web search queries.

On the other hand, researchers at Yahoo analyzed search queries and found that such searches are not always the best way to spot a trend. They looked at the volume of search queries related to a particular movie, song, and video game up to six weeks before each came out. However, the traditional critics were just as good in most cases.

It is hard to determine intent from searches and then there might be spammers in operation that compromise the data. The simple volume of searches is a fairly rough measure. The Yahoo researchers said that search data may be useful when a small improvement in prediction accuracy could have a big impact--for example, in the financial world.

The Darwin Awareness Engine™ takes a different approach to looking for trends that can be complementary to these search results.  Instead of looking at the volumes of searches, it looks at the volume of conversations about a topic.  The BuzzTape™ reports topics that are receiving rising attention in real time. The Scan Clloud™ looks at relationships between sub-themes and the main topics of interest.  Since the content self-organizes itself, it is not possible to compromise the results through SEO or spam.

 

 

Lynda Moulton Lists Five Products to Reduce Information Overload

Lynda Moulton is a consultant, an analyst with the Gilbane group, and runs the Boston KM Forum. I have attended it over the past several years. About a year ago we presented a prototype of the Darwin Awareness Engine there.  Lynda recently provided a useful post on: Focused on Unifying Content to Reduce Information Overload.  She noticed while attending the Enterprise Search Summit and KMWorld 2010 in Washington, DC last month the diversity of ways organizations are getting answers to stakeholders more quickly. I have had a chance to review her recent Gilbane Beacon report Establishing a Successful Enterprise Search Program: Five Best Practices. Here is my blog post on it (see Semantic Technologies Explained and Examined). 

Lynda noted five software products she saw at Enterprise Search Summit and KMWorld 2010. She said that, “these products, in alphabetic order, are all notable for their unique solutions tailored to different audiences of users and business requirements. All embody an approach to unifying enterprise content for a particular business function.”  The products were: Access Innovations, Attivio, Coveo, Darwin Awareness Engine, and RAMP.  I have written about Attivio and Coveo before and have great respect for them. We were very pleased to be included. Here is what Lynda said about Darwin:

“Darwin Awareness Engine was named the KMWorld Promise Award Winner for 2010. Since their founder is local to our home-base and a frequent participant in the Boston KM Forum (KMF) meetings, we are pretty happy for their official arrival on the scene and the recognition. It was just a year ago that they presented the prototype at the KMF. Our members were excited to see the tool exposing layers of news feeds to hone in on topics of interest to see what was aggregated and connected in really "real-time." Darwin content presentation is unique in that the display reveals relationships and patterns among topics in the Web 2.0 sphere that are suddenly apparent due to their visual connections in the display architecture. The public views are only an example of what a very large enterprise might reveal about its own internal communications through social tools within the organization.”

Thanks to Lynda for these comments and her post. 

 

Semantic Software Technologies Explained and Examined

I have been reading a very interesting and comprehensive report, Semantic Software Technologies: Landscape of High Value Applications for the Enterprise by Lynda Moulton of the Gilbane group. This is a topic I am both curious about and, at times, find puzzling so I was eager to see what Lynda wrote as I have great respect for her work.  I am not the only one as I often see conference sessions with such labels as “Is semantic technology real?”

Lynda wrote that she deliberately chose title of this study to refer to semantic software technologies not simply semantic search technologies. She added that the end point of any semantic software is to improve finding and interpreting content, a search activity. However, Lynda wanted to look at the total range of surrounding software tools and not simply search as narrowly defined.

The semantic tools surrounding search are often not familiar to IT and business managers so they are underutilized when opportunities for major enterprise semantic search improvements could be made. In her study, offerings that are complementary to search are examined and highlighted for their business benefits.

Lynda explained that the vision of semantic search is the availability of software algorithms that would improve retrieval for the average person by interpreting their native inquiry and returning semantically relevant results. This is as clear a definition as I have seen. In 2001 Tim Berners-Lee published an article in Scientific American proposing a semantic web evolving out of the expanding worldwide web and this concept has been seen as a holy grail for the Web ever since. 

The concept has followed the trend of other tools of moving from the Web to the enterprise. However, Lynda notes that in the enterprise, expectations for relevant search results are much higher than on the internet, where much content is already optimized for e-commerce. Each business unit in an organization has its own specialized requirements for finding information that may be in many different formats and operating under different taxonomies or none at all.

This is where other types of semantic processing beyond search can give organizations a “competitive edge by getting workers to answers more quickly, with more conceptual relevance, and even with pinpoint accuracy. The idea is to get only the right information (only relevant) and all the right information (everything that is relevant).”  Well stated.  

Her study exposes the scope and depth of software technologies that comprise the semantic tool landscape to address these issues. Topics include: text mining and text analytics, concept and entity extraction, concept analysis, natural language processing, content data normalizing, sentiment analysis, and auto-categorization. Lynda notes that these categories of technologies are very interdependent and most semantic software application offerings use several of these tools.

This is a comprehensive work that runs over 80 pages. It covers an important area that is one of the next frontiers for enterprise IT. In some ways the semantic web has run parallel to Web 2.0 and enterprise 2.0. I have even heard it described as enterprise 3.0 or Web 3.0. This is ironic since it emerged beyond the 2.0s. However, it has taken longer to become concrete. This report helps with that effort. It is available for a free download at the Gilbane site. 

 

My KM World 2010 and Enterprise Search Summit 2010 Notes: Facilitating Content Discovery Using Chaos Theory

Here is another in a series of session notes for KM World 2010 and Enterprise Search Summit 2010. In this case it is the session I did - Facilitating Content Discovery Using Chaos Theory. Here is the session description. My notes follow. 

“The presentation provides a practical application of chaos theory for creating useful content focal points from the chaotic mess of content resources distributed across the web. Several examples will be provided, both from the internet, where socially generated content is exploding, and from the enterprise, where social tools are contributing to content bloat. The examples illustrate how activity can make information more discoverable and easier to explore through various visualization techniques.”

To set the stage let’s go back to the creation of the text we use to capture content. One of the greatest information technology inventions remains the phonetic alphabet. The Greeks came up with it in the early 8th century BC. The same twenty-four letters have been used to write the Greek language ever since. It is the first and oldest alphabet to note each vowel and consonant with a separate symbol.  This invention provided great flexibility and unprecedented accuracy of linking symbol to word and thus on to thought.  Now we moved from relying on informal communication through spoken conversations to formal documents enabling many breakthroughs.

One of the great flexibilities of the phonetic alphabet or text was the wide range of vehicles that can be used to convey text. Most of the technological advances since the work of the early Greeks have been on the vehicles to convey text, such as paper over clay or stone, and the means to produce these vehicles, such as the printing press, rather than the alphabet, itself.  Text remains king on the Web and the darling of search engines.

In the past, the effort to create text acted as a filter but a series of inventions such as the printing press cut into this difficulty reducing the barriers to creating formal context, generally for the good of most.

Now with Web 2.0 the barriers to creating permanent content through text have been mostly shattered. You still need a computer or a mobile phone or some other smart device and access to the Web.  But much of what is being added is closer to the informal conversations of the pre-text world, than the type of content normally taken to print.

The economist Charles Handy noted that with the knowledge economy that knowledge workers now own the means of production, their minds, unlike their industrial age counter parts. Tings will change. Well with content production the speakers and readers how have the means of production through the Web. This has both good and bad consequences. As Plato said in reaction to a possible over-reliance on documents, conversations can clarify intensions. On the plus side we have much more to look at. One the negative side we have much more to look at.

In 2008 more content was created through the Web than in the history of content. The same thing happened in 2009.  We are now in danger of being overwhelmed by content and sinking into chaos.

The first Web search engine was created in 1990 by computer science students at McGill University in Montreal.  It's searchable web site database was limited as it did not index the content.  It is in 1994 with WebCrawler and Lycos that full-text search appeared on the WWW.  

Google added a new and transformative dimension to search with it s page rank concept. Let the readers help determine what content has the highest value and also look at currency, along with relevance   This has become very useful. I use it every day and it is the default home page for my laptop. I primarily use Google when I know what I am looking for. For example, when I want to find the score of Wizards Celtics NBA game last night or the Web Site of the Boston Celtics or an article I heard about at this conference, I turn to Google. 

However, Google Page Rank was created before the explosion of Web content and has its limits. Sometimes those better at speaking Google’s language than the others on the topic are elevated above the real experts. This is a problem when you use an external framework. People learn how to game it for good and bad reasons. SEO and spam are unintended byproducts of Google.

We do not see Chaos Theory and Correlation Rank™ as a replacement for Google. We see it as a complement. It is an awareness or discovery engine not a search engine. Instead of looking for repeatable patterns, its looks for correlations between content to discover new relationships beyond what you where initially looking for.

If you use Correlation Rank to let the content self organize there is no external framework to game or approach through legitimate methods. There is no SEO or even spam as it would serve no value. 

If I want to find the stories I did not know to look for about a topic of interest, if Ito discover new things, if I want to see the breaking news in real time, the stories generating the most buzz, I turn to Correlation Rank. It does not offer a precise answer, but it reveals a movement or trend as it occurs.  I use Correlation Rank when I want to explore any area of interest in more depth and when I want to see real time alerts.

How is it visualized? Now if I go to Google I get a linear list of results rank ordered by its external framework. Google decides what is important for you. Through the Scan Cloud™ the Darwin Awareness Engine™ provides an overview of the 100 top themes related to your topic of interest visualized in a tag cloud type structure and allows you to explore the ones that appeal to you (see below). You become the decider. In this way we see it as very similar with many of the ideas that Dave Snowden discussed in his session that I posted on yesterday. 

Through the Buzz Tape™ you see topics of rising interest.  Then you can make one of these rising topics the focus of a new Scan Cloud.

 

 

My KM World 2010 and Enterprise Search Summit 2010 Notes: Seth Grimes on Sentiment Search

Here is another in a series of session notes for KM World 2010 and Enterprise Search Summit 2010. I attended the session, Search for Sentiment led by Seth Grimes, Principal Consultant, Alta Plana Corporation. Here is the description:

“Correctly making sense of sentiment, opinion, and mood-subjective, attitudinal information-poses a special challenge for search, first in correctly indexing sentiment, then in understanding searcher intent in order to respond to search queries, and finally in presenting sentiment-search findings. In this session, Grimes will discuss applications and business benefits in areas including customer support and satisfaction, brand and reputation management, and marketing and product management. He will describe the characteristics of sentiment and opinion and outline technical approaches and challenges. He will provide examples of online sentiment analysis via the application of search technologies to attitudinal information found in traditional and social media. Grimes will also sketch a road map of likely future sentiment-search developments.”

Seth noted that sentiment analysis has been getting a lot of attention recently. Seth says his definition of enterprise search is very broad. For example, he considers Google Web search an enterprise tool because it provides a lot of value to enterprises. This makes sense.

He offered several assertions about sentiment. He first noted that when someone says it is a fact, you know it is an opinion. So he wants to be clear that these assertions are opinions. He feels that human communications are subjective and online facts and have business value. He then offer examples of facts and feelings.

He gave some examples of sentiment tools, beginning with: I Feel Fine.  It is a consumer Web tool that can be a toy. I have seen it before and agree. Then he moved to a business site that expresses opinions about travel experiences. 

Google search offers some sentiment information. This is one reason that sentiment is a search problem., because Google addresses it other search engines will also cover it.   Offering sentiment on things like travel is a decision support effort. Decision support needs to go beyond general-purpose search. For example, you need a summary of opinions.  Counting term hits does not meet the need. For example, BlogPulse does not give enough information to make decisions. It just shows mentions but you do not know why or what was said.

Seth said that sentiment analysis is the task of identifying positive and negative opinions, emotions, and evaluations. It turns attitudes into data. However, this definition is five years old and we need to go beyond positive or negative and decide what are the emotions involved.

Seth covered some tools he finds useful. The first was Jodange. It provides faceted search and the ability to do summaries. You can interact with the results. You also need to be able to assign positive and negative within components of a content segment such as a tweet.  There are often positive and negative views within the same statement, especially comparative ones.

You fall short with only doc-level analysis as there as there can be multiple sentiments in a single document. You fall short with only keyword-based analysis as you need semantic analysis. You fall short with only human only analysis as you need the power of machines to go through the vast amounts of data.  You fall short with only machine only analysis as you need the sensitivity of humans.  So you need a hybrid solution.

He showed some more examples of tools. SAS Social Media Analysis was the first and it offers social CRM. Clarabridge provides customer experience management.  Crimson Hexagon has a fine grained analysis of perceptions.  He showed an Crimson Hexagon example from Twitter political comments around the country where you can drill down by state and see the sentiment. Here is an early commentary I wrote (Essentials of Online Opinion Monitoring from Crimson Hexagon). 

Seth said that one of the trends in sentiment analysis is to go beyond polarity.  There is some academic work that is surfacing in business tools. LiveJournal offers degrees of happy, sad, and angry. This can be useful in call centers.

It seems that this is a niche in progress but some progress has been made.

 

 

 

KM World and Enterprise Search Summit 2010 Notes: Designing Social Search in the Enterprise

Here is the first in a series of session notes for KM World 2010 and Enterprise Search Summit 2010. I attended the workshop, Designing Social Search in the Enterprise. It was led by Thomas Vander Wal, Principal - InfoCloud Solutions, Inc, someone I have been following on Twitter for some time.  So I was very interested in both the topic and the presenter. Please excuse the typos as these notes were done realtime with a quick edit. Here is the session description.

“One of the biggest problem on today's websites and intranets is findability. Even as large companies have scaled up their investment in search and intranet development, one key ingredient is missing: the people. People are social, and people leverage social strategies to find information all the time-at work, with colleagues-to solve real problems. How do we add this social element into enterprise search systems? How do we know what to look for in search tools, services to add into the mix, and or design elements that will make an impact?

Our experienced designer frames these relevant models of social search and walks though how to think through the problems to get to a more optimal solution. This will also include the potential benefits as well as the pitfalls that come along with some popular solutions (as well as how to counteract those pitfalls). You will learn to think and design through the considerations to augment findability with social search within the enterprise. He will discuss the tools necessary to understand search design, the benefits of social search, and how to work though real world problem sets interactively in the session, so those in the session will come away with the ability to apply what is learned to their work environment.”

Thomas began by saying that he avoids fuzzy terms such as community, expert, collaboration, and trust, as they are hard to define.  So I guess I will not talk about building trust within communities to enable collaboration J.  I certainly agree that these terms can lead you down blind alleys. A Google search on any of the tem shows the broad range of interpretation they can generate. I especially agree about the term expert. I generally feel that anyone who calls themselves an expert, has indicated that they are not one.  

Enterprise search is problematic. It was an issue before social tools and has the potential to only get worse now with the new silos brought by some social tools. People cannot find stuff that they believe exists.  One reason is that search often lacks the feedback element. Thomas mentioned two researchers, Marti Hearst and Bynn Evans, that have worked in social search. Thomas said he used some of their framing in his session. He now went through several types of social search.

There are several types of social search – friend/colleague filtered social search. Yahoo MyWeb 2 was an example but it is no longer available. There is a problem with this approach since personal networks may be narrow.

Social ranking is another type of social search. Digg, StumbleUpon and delicious are examples. The problem is that segmentation can be difficult.

Multi-person and collaborative search is a third example. System that support shared screens, such as Google Wave, is an example.  The problem is that tools are limited. Google Wave is another tool that is no longer supported.

A fourth example is collective search such as Google News and Twitter trends. It can be used for exploratory search but the crowd is anonymous and cannot be filtered.  There are also issues around shared vocabulary. One issue that Thomas found is fear of embarrassment because you are exposing yourself by using it.

Another example is massive scale Q&A tools such as Quora. It can be useful for handling difficult search problems but the problem now is that tools are limited, especially inside the enterprise.

Thomas discussed ways to fill in the gaps in social search. He offered four types of InfoClouds. Personal InfoClouds addresse the intranet. He used this concept to cover attracting things to your screen so they are available as you work.  Then Thomas covered the concept of a Global InfoCloud, a larger set, but still within the enterprise. Then the External InfoCloud goes outside the enterprise. The Local InfoCloud completes coverage of the gaps. Information is often structured differently in specific locations such as a packy in Mass and party store in Michigan as a place to buy beer.  The Local InfoCoud can have many components such as colleagues, social software, and portals.

As you use the other InfoClouds, you place content in your personal InfoCloud. Looking back at my days within a large consulting firm I can see how these work together. My hard drive was my personal InfoCloud.  I looked to the other three for stuff to put in it, but I largely relied on personally stored content. The trouble was it was hard to search, as well as organize.

Thomas now looked at several tools, beginning with SocialCast.  It is an enterprise micro-blogging tool that I have covered a bit (see Socialcast Adds Sharepoint and Outlook Integration, New Features, and Enhanced Metrics). They have a new tool called Reach. It drops micro-blogging streams within work tools. I like this as aligning social tools with work processes is generally a critical success factor. You can also save conversations so enterprise search can find them, another good idea.

System One is another tool. It has a semantic engine that structures the information as you type. It can fill in the gaps and bring up related information. You can search as you work, It also finds relevant people based on your content.  Your entire document becomes you entire search engine. As you work, it searches and brings things up for your use.

The System One site says: “System One Radar tracks global information sources according to the context of your most important projects. Based on broadly definable Issue Portfolios it analyzes 80.000 news sources as well as Social Media, fully automated and in real-time, showing trends, backgrounds, relations, opportunities and risks. All visualized in a way that allows insight at a glance.” They have several tools and this is describing a Web version.

They also have a collaboration version. Here is its description: “System One Collaboration recombines the essentials of social software and semantic search in a highly integrated team productivity suite. Things that are being worked on are automatically related to all relevant resources within your company and beyond. System One Collaboration is browser based, works on mobile devices as well as on paper and is available as hosted or on-premise solution.”

James Robertson commented that these tools seem to attempting to make search support pervasive and come to your work. Thomas said that the trend is bringing these components to your existing tools so you do not have to learn new systems and they fit within your existing work processes.

Knowledge Plaza started as a social search engine and they added features over time (see my early review: Enterprise 2.0 Knowledge Sharing Platform from Knowledge Plaza). Thomas said he helped simplify their interface. It is a Web-based platform for enterprise search, social bookmarking, knowledge management, information brokerage and expert identification. It was developed by the Belgium based firm, Whatever. Nice name. There are facets, and you can search through other’s eyes.  You can easily package the results and share them but it does not easily integrate into enterprise search engines.

Thomas next covered reconfiguring search with social relevance. He began with Yahoo MyWeb2 which was a bookmarking tool like delicious. You could search on others bookmarks. Thomas found it very useful as people he knew where curating for him by their own curation activities. A down side was that he could not contextualize the people, especially when they discuss off topic topics.  Tagging could help here if people have the discipline to use tags. That is not a given. Some tools have auto-tagging that can help with this issue. Yahoo no longer seems to support MyWeb2. One reason is that Yahoo bought delicious.

Connectbeam is another example that unfortunately is not still around. I liked them (see Integrating Social Networking and Social Bookmarking with Enterprise Applications: Connectbeam). They believed that business networking begins around the sharing of ideas and information so they tightly integrated social bookmarking into their social networking platform. They also introduced the Connectbeam web services Application Programming Interface (API) that enables you to add full functionality of Connectbeam social software into your existing IT applications. Perhaps they were a bit ahead of their time.

Thomas next covered search across walled gardens. Enterprises create different levels of openness.  Search needs to take this into consideration and honor the layers of accessibility.  

Future search and follow results was the next topic. PubSub was an example of this but it is no longer around. You could get alerts on topics of interest for advanced business intelligence. Perhaps it was another company ahead of its time. 

Thomas next discussed the concept of social comfort. There are three components: social comfort with others, social comfort with tools (e.g. How far is the reach of my actions?) and social comfort with subject matter (where can I talk about certain topics?). These are issues to resolve as you use social tools. 

There was a lot of useful information and context in the session. Thomas has a depth of experience here. It seems that this is still a field that is a 'work in progress." Many of the initial tools are not around any more. Others are still being refined. One experimental tool in this space that I like is IBM Social Lens (see IBM’s Social Lens Provides Smart Content Filter).  I look forward to seeing what progress is conveyed in next year's session.

 

 

KM World and Enterprise Search Summit 2010 Coming to Washington DC on Nov. 15 - 18

KM World and Enterprise Search Summit 2010 are around the corner. It is one of the longest running shows in the business event world.  It is both a testament to the long running nature of knowledge management and the value these combined events bring to that movement and enterprise IT in general.  I am looking forward to attending this event as i did last year. It will be held at the Renaissance Washington DC hotel. There is another great line up of speakers. I will be writing about many of the session on this blog.

I am also pleased to be speaking on Facilitating Content Discovery Using Chaos Theory as part of theKnowledge Showcase. My session is from 12:30 to 12:45 on Thursday. They are taking place at the KM Corner in the conference Showcase, these presentations cover a range of topics & subject areas. Here is the session description.

“The presentation provides a practical application of chaos theory for creating useful content focal points from the chaotic mess of content resources distributed across the web. Several examples will be provided, both from the internet, where socially generated content is exploding, and from the enterprise, where social tools are contributing to content bloat. The examples illustrate how activity can make information more discoverable and easier to explore through various visualization techniques.”

 

 

 

Coveo Customer Information Access Solutions (CIAS) Moves to Version 2.0

Coveo now has a three-part product strategy. First, there is the core search platform, Coveo Enterprise Search 6.2, the company’s enterprise-grade information access solution, used by the likes of Lockheed Martin, GEICO and YUM! Brands; next there is Coveo Expresso (see my post: Coveo Provides Free Entry Level Enterprise Search With Coveo Expresso) for small and mid-sized businesses, and third there is its Customer Information Access Solutions, used by organizations with knowledge-intensive customer support needs, like CA Technologies and Quantum. The latter was first launched in October 2009 and version 2.0 came out this summer.  I recently spoke with Louis Tetu, Rich Tessier, and Diane Berry, of Coveo about this comprehensive offering.

The Customer Information Access Solutions (CIAS) are a set of individual, but related, extensions of the core platform that address specific business issues.  They are built with concept of “stop moving data” in mind. Companies spend large sums of money and too much time trying to take data from multiple sources and attempting to put it in one place for easier access. With Coveo’s unified index approach you do not need to move this data since it can be accessed from its original location.  This can include emails, documents, CRM apps, databases, knowledge bases, social media, Web content, and more. 

There are three interrelated concepts here that I like: focused business apps built on search, single point of information access, and the dynamic mashup of information to fit specific user needs. These are part of the foundation for an effective enterprise 2.0.

We first discussed CIAS for Customer 360 that provides analytics dashboards. It provides comprehensive account information in a single view, bringing in relevant data from multiple sources. Coveo comes with many out-of-the-box connecters to sources such as Sharepoint, Salesforce.com, Jive, and Atlassian. Others can be quickly created if necessary.  These content sources are combined into a single index for unified access. This allows the user to easily refine your searches and narrow the target data set rather than having to operate within the constraints of the data architecture. In addition you can consolidate and correlate content through mashups and monitor content to get up with essential information. Below is a screen shot of customer account metrics.

In addition, there is Coveo CIAS for Contact Centers that provides agent support through the same core capabilities in Customer 360 including unified access. Often contact center agents have to sort through multiple systems with different search capabilities and interfaces. I remember being told by an agent that one of the core skills was engaging in entertaining chat with the customer while they waded through the many sources to find answers. Below is a sample CSR screen.

Coveo CIAS also includes Customer Self-Service with a similar robust set of capabilities. The interface and content is a subset of the contact center application to achieve consistency, another move I like. There is contextually relevant guided navigation to reduce drop-offs and increase self-service.

To further support monitoring and continuous improvement there is Coveo Usage Analytics that compiles and presents all user interactions with the CIAS solution. You can see top queries, documents, users, and various user behaviors. Managers can see trends content gaps, and other useful insights. You can see a sample screen from the Usage Analytics below.

Louis mentioned the customer example of Computer Associates that has rolled out the Coveo platform to its 1600 agents. Since then CA has achieved a 10% increase in customer self-service satisfaction, a 50% reduction in case resolution time, and they have deflected 10% more calls to self-service, hitting about 50,000 queries a day with a response time of 0.2 seconds. I can believe these results as I have seen similar effects when contact center support tools are enhanced.

Companies first implemented knowledge base solutions to assist contact center agents. These offered some help and I was involved in a number of them. However, it required the creation of yet again another information source for agents to content with and it significantly increased the development time and cost.  Now this effort can be bypassed with a unified access solution such as Coveo CIAS.  This is truly a break through in information access. I think that Coveo was right to target contact center support since the ROI here is huge as the CA results demonstrate. At the same time there are many other applications that can benefit from this approach. It is at the core of enterprise 2.0. 

Twortex Provides Twitter Search through Darwin Ecosystem’s Awareness Engine™

It can be addictive. Twortex is a Montreal based company that offers a free Twitter search engine powered by the Darwin Awareness Engine™.  I now use it to discover new conversations on Twitter and sometimes find items related to my topics of interest that I did not anticipate were connected to these topics. The Darwin Scan Cloud™ provides the top hundred themes connected to an attractor, our term for a search topic of interest. As you scan over it you can see related themes. Here is a Twortex Scan Cloud™ covering the term – enterprise20 – for the past 24 hours on 9/10. The actual tweets connected to these terms are seen on the right of the screen. Clicking on the listed tweet can take you to the actual tweet.

 Here is the search on tweets for the past 120 hours on 9/10 on the same term – enterprise20 – for the past 120 hours on 9/10. You can see the changes on coverage.

In the screen below you can see some interrelated terms highlighted in red after mousing over one of the terms in the scan cloud shown above.  This provides an alternative to traditional Twitter search that presents topics in a liner order that can make it harder to see trends and more difficult to quickly scan over a large amount of content. You can go to the Twortex site and register and to use it at no cost.  

 
 

One Take on the Gory Detail on How Google Works

Recently the PPC blog posted an useful diagram giving their take on How Google Works: in Gory Detail.  It is interesting that they choose to display the information in a visual that does not have searchable text like a traditional blog post.  Where they trying to hide their content from Google?  They do say that Google is very secretive about their data centers. 

Gavin Heaton who writes the Servant of Chaos blog wrote a nice commentary on the PPC work on MyVenturePad, which he titles The Scary Truth About Search.  He notes the major search optimization that many sites use and then concludes with the thought that web sites need to think about what happens when you “throw “social” into the mix and see how you stack up. Scared? You should be.”  Google Page Rank was created before social media. Has this social media compromised Google’s approach? 

eDiscovery Integration Remains a Challenge Despite More Market Consolidation

According to Forrester's Brian Hill, recent mergers and acquisitions in the eDiscovery market can benefit organizations wanting to mitigate legal risk with a mix of disjointed applications as long as they do their home work (see eDiscovery Market Consolidation Continues Its Steady March). As usual, organizations must separate marketing hype from actual functionality, especially in the area of end-to-end process capabilities. While integrated advances can provide concrete benefits and help rationalize application infrastructure, it's important to look at these in the context of a broader eDiscovery strategy.

Such strategies should identify technology gaps and costly process integration points. Then enterprises need to request more eDiscovery application integration. Even then you cannot expect to end up with a single provider.  The report notes that recent survey results indicate that 60% of records management stakeholders and 57% of message archiving users perceive “synchronizing eDiscovery, records management, and archiving efforts” to be a challenge.

In theory the transparency within enterprise 2.0 should support greater eDiscovery capabilities but there is still the tendency to create even more silos. Cross platform capabilities need tor receive a greater focus.  The report offers some useful guidelines for enterprises as they navigate this space.

 

Workstreamer Enables Web Listening for a Broader Audience

There are a number of ways to monitor the web to see what and how you and others are doing. Many of these require more configuration than the average business user is willing to engage in.  Workstreamer is designed to address the need of users such as sales and customer service reps, supply chain managers, and others who what to easily track what is occurring and what is being said on the Web about specific companies.

I recently spoke with Hank Weghorst, Co-Founder and CEO and Suaad Sait, Co-Founder, about their offering. It is described as a "real-time business listening platform" that allows you to stop searching and start listening. They automate the listening function for tracking business information, not just social mentions of your brand.  So it brings in data from blogs, tweets, social networks like LinkedIn, contact directories like Jigsaw, finance sites, CRM services like Salesforce, and more. The results are presented in a stream of findings that can be expanded to drill down on specific companies. You can see a sample screen below.

Looking at the screen above from left to right you first see the user and the companies they are following.  Then you see the feed of detail as around each company. The waving line shows the varying amount of buzz. The numbers indicate the change in mentions from the day before. Then the company details are offered with icons linking to such sources as Facebook, Twitter, YouTube, LinkedIn, Jigsaw (the business card site). The next column provides what they call the tone cloud. It is a tag cloud like visualization that shows the key themes emerging for each company and allows you to drill down to the specific mentions.

In the far right column you can see the other team members. Workstreamer allows the formation of groups to share information. They start with others in the same domain but then others can be added.  This team feature comes in handy as you can comment on results and these comments are shared with team members. For example, a sales manager can point out the significance of news items to the rest of the team or ask someone to follow up on a change within a client.  You can see a comment in the screen below

 

One of the things I especially liked was the capability of expanding the results through specific visualizations. For example, the numbers of job openings over time is presented in a line graph.  Details on executives from Jigsaw are displayed in a table. Each visualization is specific to the type of data covered. Below you can see a graph of job posting by a company in the midst of the stream of other news items. 

It is very easy to start tracking a company. Workstreamer has already added over million companies into their database and they continue to add more. So if you type in a name, they offer you instances to choose from. If the name is not in their database yet you can still start tracking it. I gave them the name of a very small local company and they came up with it instantly.  The goal is a one-click start and this appears very possible. 

Workstreamer does a lot of filtering for business relevance so you are not overwhelmed by useless information. You can also select to narrow results to specific content types such as news sites or social media.  In addition, you can connect Workstreamer to your LinkedIn and Salesforce.com accounts. They provide nightly email summaries of the top news in the firms you are tracking.

The current version of Workstreamer is free. They plan to remain free through 2010 to build awareness. The free version will always be available and always retain its features. In 2011 they plan to offer a fee-based premium version with additional features. For what I have seen the free version is quite robust and useful.  I am very impressed with both the ease of use and the usefulness of the data visualizations. I plan to give it a try myself. 

Enterprise Search Summit in Washington DC for Fall 2010

I plan to attend this the Enterprise Search Summit this November in Washington DC. I was at the last one in NYC in May and the one before in Santa Clara in November 2009. It is a great conference and I look forward to the next one. This session will meet at the same time and place as KM World 2010, just as did last year in November. I will be moderating a day's set of sessions at KM World and writing about both conferences. 

My Darwin colleague Thierry Hubert and I also enjoyed the May 11 - 12 Enterprise Search Summit held in New York. We presented on presented a session, Deriving Order From Chaos Through Discovery and Awareness.  Here is a series of posts that covered some of the points we made in the session (Rising Above the Over Quantification of Content). Here also is a summary of my Enterprise Search Summit 2010 notes on other sessions I attended. 

The organizers at Enterprise Search Summit have give me a $200 discount to pass on to you. Use this link to sign up. Hope to see you there. 

How Will Search and Social Media Converge?

I read an interesting interview recently. Steve Rubel is senior VP-director of insights at Edelman Digital discussed the convergence of search and social media. He makes a nice distinction between the two. “Search is an intent-driven medium, where users seek out what they want. Social networking is where the content finds you through the lens of friends. Those two are separate, but I think we'll see a real convergence here where search will get a lot more social and social will get a lot more searchy.”

You can now search your friend’s content. For example, with Google you can bring in the results from your social connections via the Google address book. At the same time you can go into Facebook and search your friends' content.  As a result Steve predicts that push and pull will merge. We will become more media-agnostic through an aggregator like Google News, along with a social stream, where we go in and out.

I like this prediction as the Darwin Awareness Engine (TM) can serve the same aggregation function as Google News but with more robust features. The Awareness Engine allows you to spot trends through a combination of content, including your friends’ content or their content choices.  It will be interesting to see where this goes. 

Using Chaos Theory Principles to Overcome Information Overload within the Enterprise and on the Web – Part Two - My Enterprise 2.0 Conference Notes

This is the tenth in a series of notes on the Enterprise 2.0 conference in Boston, June 14- 17. This one is different as it covers the session led by myself and by my Darwin colleague, Thierry Hubert, Using Chaos Theory Principles to Overcome Information Overload within the Enterprise and on the Web.  Here is a link to the notes on part one of this session. Here is our description and summary of Thierry’s comments follows.

“With the addition of social media to the already increasing amount of Web content companies are spending more resources trying to make sense of what is happening. In addition, the connections between formal, informal, structured and unstructured information are becoming more difficult to establish. It may only get worse with the advent of auto-generated content and other content “farms.” In this session we will explore and discuss how the application of chaos theory to this issue can help break the silos of information and allow the emergence of meaningful awareness for better decision making.  This new paradigm moves beyond Page Rank to reduce the impact of traditional SEO techniques that elevate low quality content.  We will look at how the visualization of connections between content related to a particular theme can reveal new relationships and help with the discovery and awareness of trends, both anticipated and unanticipated.”

Thierry began but pointing out how we are surrounded by chaos on the Web through the over abundance of content. Page rank was a big improvement over prior search methods. However, it was conceived before the advent of social media. The increased and growing volume of Web 2.0 information is causing search engines to rethink their outdated popularity ranking algorithms by bringing timely social media content to the foreground in an attempt to increase relevance for users. 

In the recent FaceBook privacy controversy, Web users made it clear that they are not so eager to allow search engines to mine their communities’ content for the benefit of search engine companies and other third parties.  As a result, social media content is not rescuing search engine companies’ quest for intimate relevance.   Nonetheless, search is a critical part of the Web experience, but it is no longer the only way to deliver relevant information.

It is since the advent of the Web 2.0 that search engines are no longer capable of elevating recent and relevant information in a timely and contextual manner.  Popularity ranking is corrupted by Search Engine Optimizations (SEO) and requires too much time to elevate pertinent information that has a short shelf-life.  The sad thing is that users are missing meaningful information that is hidden on the 50th-plus page of a search result; they just don’t know what they don’t know since they can’t find it.

To address this challenge Darwin Ecosystem’s Awareness Engine™ applies chaos theory principles to reveal emerging patterns and correlations that accelerate the user’s understanding of what is happening about a given theme of interest regardless of popularity ranking.  We believe this organic and self-organizing model is best adapted to the evolution of today’s Web when compared with the increasing limitation of page ranking offered by search engines.  Although not a replacement for search engines, Darwin’s Awareness Engine highlights a new and missing perspective. Chaos theory does not offer a precise answer, but it reveals a movement or trend as it emerges in real time.

Thierry provided a number of examples of Chaos Theory in operation through the Darwin Awareness Engine. The Darwin interface provides a simple and interactive ScanCloud™(patent pending) that reveals correlated themes for contextual content filtering around your topics of interest, a BuzzTape™ that displays themes that have high content-acceleration within the last 24 hours, and the ability to save attractors (term used in chaos theory) to monitor the evolution of your favorite topics. Below is a sample Scan Cloud.

At the top is the Buzz Tracker ™showing the rising and falling starts. At the top left you can see that it is set to catch what is happening in the past hour. The Scan Cloud ™ shows the top themes emerging in the target sites. The right column shows the stories under these themes with links to the actual story. Now here is a close up of the Scan cloud so you read it better.

In this instance there is a reference to Knoxville in the middle of the Scan Cloud. If you click on it the stories in the right column shift to those related to Knoxville and display two stories about use of social media related to the recent floods in the area.

Looking at what was under Twitter in the Scan Cloud and the two stories below are found among others.

You can link to one of the stories in PC Magazine as displayed below.

Thierry provided a number of examples but this one gives a sample of the basic interface and process. Darwin’s Awareness Engine can provide a new model of information consumption through Chaos Theory that will benefit Web and Enterprise users alike while allowing Web 2.0 authors to be noticed in a timely and relevant context. 

Using Chaos Theory Principles to Overcome Information Overload within the Enterprise and on the Web – Part One: My Enterprise 2.0 Conference Notes

This is the ninth in a series of my notes on the Enterprise 2.0 conference in Boston, June 14- 17. This one is different as it covers the session led by myself and by my Darwin colleague, Thierry Hubert, Using Chaos Theory Principles to Overcome Information Overload within the Enterprise and on the Web.  Here is our description and then summary of my opening comments follows. The next post will cover Thierry’s comments.

“With the addition of social media to the already increasing amount of Web content companies are spending more resources trying to make sense of what is happening. In addition, the connections between formal, informal, structured and unstructured information are becoming more difficult to establish. It may only get worse with the advent of auto-generated content and other content “farms.”

In this session we will explore and discuss how the application of chaos theory to this issue can help break the silos of information and allow the emergence of meaningful awareness for better decision making. 

This new paradigm moves beyond Page Rank to reduce the impact of traditional SEO techniques that elevate low quality content.  We will look at how the visualization of connections between content related to a particular theme can reveal new relationships and help with the discovery and awareness of trends, both anticipated and unanticipated.”

I started by saying that we do not see Chaos Theory as put in practice through our Darwin Awareness Engine™ as a replacement for Google. We see it as a complement. It is an awareness or discovery engine not a search engine.

For example, when I want to find the TV schedule for the NBA playoffs or the Web Site of the Boston Celtics or even the Los Angeles Lakers, or an article I heard about at this conference, I turn to Google.  I use Google when I know what I am looking for.

If I want to find the stories I did not know to look for about a topic of interest, if I want to discover new things, if I want to see the breaking news in real time, the stories generating the most buzz, I turn to Darwin. Chaos theory does not offer a precise answer, but it reveals a movement or trend as it occurs.  I use Darwin when I want to explore any area of interest in more depth.

Now if I go to Google I get a linear list of results rank ordered by its external framework. Google decides what is important for you. Darwin provides an overview of the 100 top themes related to your topic of interest visualized in a tag cloud type structure and allows you to explore the ones that appeal to you. You become the decider and we will show some real time examples.

Rather than taking a linear, deterministic approach to finding content through seeking repeatable patterns on the Web with an external framework like page rank as done by Google, we use Chaos Theory to find non-repeatable patterns in Web content that come and go in real time. Rather than using an external framework, we look for self-organizing patterns within the content itself.

Sometimes external frameworks and measurement have their place. Other times it can be an obsession that gets in the way. For example, Fred Taylor tired to systemize work processes and this approach led to profits in some cases but has alienated most workers. In his Atlantic article, Will Google Make Us Stupid, Nick Carr quoted Google CEO Eric Schmidt, that Google is “a company that’s founded around the science of measurement,” and it is striving to “systematize everything” in the same spirit as Taylor. This can be useful; but it also has its limits. It depends on the use case. Robert Parker has tried to systemize wine reviews and in my view is a clear over step. How can you say with any certainly that one wine is a 94 while another is a 92 and be serious about it?

We offered this video showing Robin Williams in The Dead Poet’s Society discussing the value of a measurement system for poetry. Here is a link to the Measuring Poetry episode on YouTube.

Now I am not saying that Google is like Pritchard on poetry or Robert Parker on wine in that it is useless. Often you have to lay pipe in Robin Williams terms. It is the default home page on my laptop and I use it everyday. However, I do say that its quest to systematize everything and make it serve all content discovery functions is a delusion. 

 Google has its limits. For example if I type in - Sardinian beach bars - into Google I find that my blog post is number one out of 349,000 as it has been for the past five years since I wrote it. I have even used external frameworks myself.

On the same trip my friend and I where sitting at one of these beach bars and over a few glasses of wine devised a measurement system to rate the beaches of Italy. Now if you type rating the beaches of Italy into Google you will find that my blog posts are number one and two out of 691,000.   

Of course I offer these examples with great humility. I have no delusion that I offer the best content here or that I am the world’s expert. I was just better at speaking Google’s language than the others on the topic.   This is a problem when you use an external framework. People learn how to game it for good and bad reasons. SEO and spam are byproducts of Google.

I must confess that I have done some consulting over the past three years helping firms get their blogs on the front page of Google, using respectable methods I add.

If you use chaos theory to let the content self organize there is no external framework to game or approach through legitimate methods. There is no SEO or even spam as it would serve no value. 

To set the stage we went back to the creation of the text we use to create content. One of the greatest information technology inventions remains the phonetic alphabet. The Greeks came up with it in the early 8th century BC. The same twenty-four letters have been used to write the Greek language ever since. It is the first and oldest alphabet to note each vowel and consonant with a separate symbol.  This invention provided great flexibility and unprecedented accuracy of linking symbol to word and thus on to thought.  Now we moved from relying on informal communication through conversations to formal documents.

One of the great flexibilities of the phonetic alphabet or text was the wide range of vehicles that can be used to convey text. Most of the technological advances since the work of the early Greeks have been on the vehicles to convey text, such as paper, and the means to produce these vehicles, such as the printing press, rather than the alphabet, itself.  Text remains king on the Web and the darling of search engines.

In the past, the effort to create text acted as a filter but a series of inventions such as the printing press cut into this difficulty reducing the barriers to creating formal context, generally for the good of most

Now with Web 2.0 the barriers to creating permanent content through text have been mostly shattered. You still need a computer or a mobile phone or some other electric device and access to the Web.  But much of what is being added is closer to the informal conversations of the pre-text world, than the type of content normally taken to print. This has both good and bad consequences. As Plato said in reaction to a possible over-reliance on documents, conversations can clarify intensions. On the plus side we have much more to look at. One the negative side we have much more to look at.

Next I showed the evolution of content delivery channels.  In 2008 more content was created through the Web than in the history of content. The same thing happened in 2009.  We are now in danger of being overwhelmed by content and sinking into chaos. Why not turn to Chaos Theory to help pull us out? Instead of imposing an external framework to attempt to make sense of content, why not let content self-organize itself. Instead of looking for repeatable patterns, why not look for correlations between content to discover new relationships beyond what you where initially looking for?

My colleague Thierry Hubert now took over to explain how this can work.  His comments will be in tomorrow’s post. 

Enterprise Search Summit Moves to Washington DC for Fall 2010

The Fall Enterprise Search Summit has moved to Washington DC on November 16 – 18 along with KM World. It will be at the Renaissance Washington, DC Hotel. I enjoyed last year’s joint session and wrote a bit about it (see for example My Enterprise Search Summit Notes: The Role of Social Techniques in Search and How It Impacts Your Organization). 

My Darwin colleague Thierry Hubert and I also enjoyed the May 11 - 12 Enterprise Search Summit held in New York. We presented on presented a session, Deriving Order From Chaos Through Discovery and Awareness.  Here is a series of posts that covered some of the points we made in the session (Rising Above the Over Quantification of Content). Here also is a summary of my Enterprise Search Summit 2010 notes on other sessions I attended. 

I hope to attend the combined Enterprise Search Summit and KM World in DC and hope to se you there. Registration is open at their site. 

. 

Rising Above the Over Quantification of Content

I recently posted a four part series on the Darwin Discovery Engine Blog on Rising Above the Over Quantification of Content. I will not repeat the whole series here. Below are links to the first three posts. Then I offer the concluding post, Rising Above the Quantification of Content: Part Four Google vs. Darwin. I encourage you to look at the first three before moving to this final episode. 

Rising Above the Quantification of Content: Part One: Parker vs. Piaget

Rising Above the Quantification of Content: Part Two: Pritchard vs. Williams (aka Keating)

Rising Above the Quantification of Content: Part Three: Powerpoint versus Text and Conversation

In this last part of the series on the value of providing people the opportunity to exercise their own cognitive powers to make sense of content, see the connections between content, and decide what is valuable to them. I have reviewed some efforts to over measure topics beginning with Robert Parker on wine and J. Evans Pritchard on poetry and looked some effects of media on cognition by exploring what PowerPoint may be doing to us as reported in the New York Times.  Nick Carr addresses similar questions in his Atlantic article, Is Google Making Us Stupid? He provides some anecdotal evidence but admits that the proper scientific research has not been done.

I do not feel that Google actually makes us stupid but also do not feel that it is not going to be the key to understanding the universe that seems to be the quest for Google.  I am also not suggesting that Darwin is here to replace Google. It simply complements it and does things that Google is not so good at. Google is a search engine and it helps you find things you are looking for. I keep using it to write this post. Darwin is a discovery engine and it helps you explore a topic and develop a greater awareness of its context and uncover things about the topic that you did not know to search for. I have written about the Google question before (see Will Google Make Us Stupid? in the FastForward blog). I concluded that like any information tool, the final responsibility for determining the value of the content lies with the person using it. Darwin is designed to give the user a greater opportunity to make these decisions.

I do agree with Nick Carr that media and knowledge related technology effect cognition, that the means we use to work with information have effects on our thought process. I underplayed this aspect of the Google question in my FastForward post. Our communication channels, whether they are words, charts, PowerPoint, or Twitter, provide structure to our thoughts and help shape them and our subsequent thinking. This was the focus of my psychological research in the 70s, first at the University of Toronto and then at Harvard.

One clear example of the effects of media on cognition comes from comparing spoken language with sign language. For example while errors in speech often reflect the structure of spoken language, the neuroscientist Ursula Bellugi found that errors in sign language reflect its parameters: space, direction, and movement. Likewise the poetry of spoken language reflects the rhythms of sound while the poetry of sign language is based on the rhythms of visual movement. A communication medium does more than convey information. It helps to structure your thoughts.  In the last post in this series I discussed a bit about the effects of the phonetic alphabet on thought.

Carr offers a clear example of the effects of knowledge related technology when he discusses how the mechanical clock, which came into common use in the 14th century, changed aspects of thought. He refers to the work of Lewis Mumford, the historian and cultural critic who described how the clock “disassociated time from human events and helped create the belief in an independent world of mathematically measurable sequences.” The “abstract framework of divided time” became “the point of reference for both action and thought.”

So both technology and communication media can effect thought. The Web is a mixture of both. Carr wonders about the effects of using the Internet on our cognitive abilities. He writes, “Never has a communications system played so many roles in our lives—or exerted such broad influence over our thoughts—as the Internet does today. Yet, for all that’s been written about the Net, there’s been little consideration of how, exactly, it’s reprogramming us.”

Carr considers Google to be the “Internet’s high church” and he goes into great depth on how Google’s obsession is with measurement, like Fred Taylor, Robert Parker, and J. Evans Pritchard. He quotes the Google CEO Eric Schmidt, that Google is “a company that’s founded around the science of measurement,” and it is striving to “systematize everything.”  Sound familiar?

Carr goes on to write, “It seeks to develop “the perfect search engine,” which it defines as something that “understands exactly what you mean and gives you back exactly what you want.” In Google’s view, information is a kind of commodity, a utilitarian resource that can be mined and processed with industrial efficiency. The more pieces of information we can “access” and the faster we can extract their gist, the more productive we become as thinkers.” 

He also notes that the economic model of massive page views drives them to encourage us to skim from page to page, not stopping to explore and contemplate.  Google provides a linear list of results organizer according to Google’s concept of what is useful to us. Google tries to be the decider on value. Carr is concerned that this is causing us to move away from reflective thought and focused reading for a deeper meaning.

Darwin takes an opposite view. It views the Web as too massive to understand and human reasoning too complex and individualized to fully progam. Rather that drawing inspiration for Fred Taylor’s time and motion studies and attempting to understand and control the universe of content on the Web, Darwin is inspired by Edward Norton Lorenz’s work on Chaos Theory in mathematics and weather prediction. Darwin does not try to impose an order on content, it lets content self-organize. During the 1950s, Lorenz became skeptical of the appropriateness of the linear statistical models in meteorology, as most atmospheric phenomena involved in weather forecasting are non-linear. Instead he proposed the concept of attractors that operate within a dynamic system that evolves over time in a complex, non-repeating pattern.  This is the complete opposite of Fred Taylor’s repeatable time and motion approach.

We think the two approaches complement each other. The Darwin approach does not require human intervention to be organized; it is self-organizing based on the content itself and therefore more representative of what is happening as it occurs, and respectful of each unique context. It also eliminates the possibilities of search optimization and spam.  Instead of providing a linear list, Darwin offers a Scan Cloud of the top 100 themes related to your topic of interest (see below).  Your chosen topic acts as an attractor to collect related content. Darwin allows you to move within the Scan Cloud to explore the related themes that are of interest to you and the Scan Cloud dynamically changes to reflect your movement. 

A list of the actual content within each theme is found next to the Scan Cloud (see below) and it dynamically changes to reflect your movements.  Darwin does not try to understand the universe of content nor does it try to understand your intentions or values. Instead it helps you to achieve your own understanding of what is happening in the world of Web content.

Darwin promotes exploration, contemplation, and discovery. We do not make any claims on its lasting impact on thought but it does address some of the concerns that Nick Carr raises. We invite you to contact us to take a look at our Darwin web site. 

 

Summary of My Enterprise Search Summit 2010 Notes

I recently attended the 2010 Enterprise Search Summit in New York. It was a very useful conference. My Darwin partner Thierry Hubert and I presented a session, Deriving Order From Chaos Through Discovery and Awareness.  Here are my notes from other sessions. I hope they are useful.

Enterprise Search Summit 2010 Notes: Search for Customer Satisfaction at Standard & Poor’s

Enterprise Search Summit 2010 Notes: Improving Findability Inside the Firewall

Enterprise Search Summit 2010 Notes: Sharpening Enterprise Search Performance and More

Enterprise Search Summit 2010 Notes: Search and Discovery Patterns

Enterprise Search Summit 2010 Notes: BI in the Age of Social Media

Clearwell Offers Comprehensive Process Aligned eDiscovery Solution

Before there was enterprise 2.0 I used to think that, except for utilities like email, most business information-based software such as knowledge management needed to be process aligned. I have softened that position somewhat but for certain tasks, process-aligned remains the key to effectiveness. This is the approach taken by the Clearwell E-Discovery Platform, a product targeted at information-related legal process such as discovery. I recently spoke to Kamal Shah, the Clearwell VP of Products and Marketing, on how this product provides end-to-end process support for – early case analysis, search, cull, review, redaction and production.

Clearwell was founded in 2004 and they are about to release version 5.5 of their eDiscovery product. They now have over 500 customers including Cisco, Constellation Energy, Microsoft, BP and NBC, and they work with 42 business partners. They offer the software directly to enterprises and also sell through business partners who provide Clearwell as a hosted service.  Clearwell provides an appliance based solution that allows for rapid deployment, often in about 30 minutes.

Kamal explained that the industry has adopted an Electronic Discovery Reference Model to promote a common vocabulary and other standards within the space.  Clearwell supports this model and provides a common platform for the various stakeholders in the eDiscovery process including legal, IT, HR, audit, security, and compliance.

Kamal went on to describe the eDiscovery process and how Clearwell supports it. First, there is the initial data collection. Once data is collected, then Clearwell provides pre-processing capabilities. At this stage you also want to understand the task at hand and effectively estimate the cost and duration. Clearwell is able to go after multiple data types and sources to ensure effective retrieval.

 Figure 1: Timeline Analytics Quickly Identify Gaps in Collection Across All Data Sources

Next comes the processing. Clearwell has ramped up their processing speed to help with this effort. With the new 5.5 Release, Clearwell can process data at speeds of over 1 terabyte per day and scales to 100 million documents on a single appliance. De-duplication also occurs in this step as you want to reduce the amount of information that your legal counsel, operating at high hourly rates, has to sift through. Part of this is eliminating copies of redundant content such as duplicate files and multiple copies of email forwarded around. At the same time you want to know who knew what and when in these email chains. Clearwell sets them up as a threaded discussion to easily uncover the answers to these questions.

 
Figure 2: Discussion Threads Identify All Participants and Who Knew What and When

Now comes search and cull down to further refine the content. Usually only 10 to 20% of the content is relevant so you want to get there as efficiently as possible.  You can set up auto-filters to eliminate known false positives such as an article on the topic that was forwarded around but not relevant to the case. Next step is search. A firm might get several hundred keywords  from a government agency such as the SEC or from an opposing litigator. The trick is to provide all of the relevant information and no more. If you provide too much you may be alerting the opposition to potentially damaging information not relevant to the inquiries. Providing too little may result in damaging later discoveries and penalties for failing to properly comply.  Clearwell provides robust keyword and concept search capabilities that help you fulfill the request quickly and easily.

Figure 3: Transparent Search Reduces False Positives of Keyword Search Results

After distilling down the content, Clearwell supports the legal review process with nested folders, decision tree tagging, and indicated related documents. There is a simplified review interface to speed the process. Finally, the post-review production phase is supported through such features as integrated production folders, image based production, custom labeling using headers, footers, and watermarks.

 
Figure 4: Review Capabilities Allow Users to Quickly Redact Privileged or Confidential Data

Kamal said that the cumulative effect of these process-aligned supports can result in significant costs savings.

Because of the ease of implementation, Clearwell lets potential clients set up test cases to determine their own ROI and this usually results in a new client. I can see why.  It certainly makes to sense to use a product like Clearwell in your enterprise legal processes. This is one situation where process aligned remains a requirement for success. 

Gary Flake Demos Pivot at TED

Gary Flake recently demoed Pivot at TED.  It is a new way to browse and arrange massive amounts of images and data online. Built on Seadragon technology, it enables spectacular zooms in and out of web databases, and the discovery of patterns and links invisible in standard web browsing. Gary is a Technical Fellow at Microsoft, and the founder and director of Live Lab. He shows how you can take control over data to see trends that might be undiscovered otherwise.  For example, being able to see all Wikipedia pages related to a topic at once, allow you to see trends that are not contained in any single entry.

He said that with Pivot you can navigate the Web as if it is a Web and see relationships.  He talks about taking the curse of data overload and turning it on its head to take advantage of quantity to see the patterns within it. Here is Gary’s Pivot talk on TED.

Although we take a different approach this is exactly the goal of the Darwin Awareness Engine (TM). Instead of bring you long lists of content on topic that are order by popularity, we show the top 100 themes within the content that relates to your topic of interest.  We attempt to let you discover unexpected patterns, content, and relationships in this manner. 

2010: Social Media Removes Gutenberg-Google Content Dam

Here is an interesting post that I can certainly agree with. Aaron Kahlow writes about the information curation potential of social media in his post, 2010: Social Media Removes the Dam of Gutenberg-Google.

Aaron begins with the effect of the printing press. It certainly opened up content distribution way beyond what the monks doing hand written work could accomplish. However, it established new controls over what content got out. You still needed to have the resources to set up a printing press, print and distribute works.

Now there is the Web and you might think the dam is broken. In one sense it has, as there are very few barriers to getting content out on the Web. In 2008 there was more content created than in the entire prior history.  The trick is finding this content, especially the quality stuff. This is where Google has both helped and created a new bottleneck. As Aaron writes, “With only 10+ organic results and a similar number of paid results, consumers will only find content they seek in those 20 places -- with less than 20 percent being relevant as it relates to information they seek.” For example a search on “green card” will get you a lot of services that want to charge you money for what the US government does for free.

Aaron goes on to comment that social media can help break this new dam. Now we can “get good information through tweets of those we follow, Facebook Sharing, and from others within our networks who are usually connected online. We have a new discovery outlet and a new way to find stories, whether mainstream or from an unknown blogger. We find things based on recommendations of trusted colleagues, friends, etc.”  This is how I found Aaron’s post.

Peter Cashmore raises a similar point in his predictions for 2010 than I commented on earlier (see Reflecting on Peter Cashmore's Web Trends to Watch in 2010). Peter wrote that, “The Web's biggest challenge of recent years is that content creation is outpacing our ability to consume it: "Information overload" has become an increasingly common complaint… In 2008, the answer revealed itself: Your friends are your filter… Increasingly, your friends are becoming the curators of your consumption.”  I certainly agree here and Twitter has served this role for me.  Much of what I write about on my blogs comes from my Twitter friends, including the link to Peter Cashmore’s predictions.

Aaron offers some good points to make effective use fo social media to get your content out. Instead of having to contact a prominent journalist to write a story about your company, you can use your own channels such as blogs to consistently write good content worth tweeting and sharing.
 You can also leverage your own social networks to start sharing content. 
These are all reasons that we write this blog and participate in Twitter and other social media.

Darwin Ecosystem Awareness Engine™ is designed to address this issue of finding relevant quality content without having to go through the filter of Google or other search engines. Unlike Google, it does not decide what content is most relevant but rather lays out the content related to your topic of interest in clusters of themes and lets you explore what themes interest you. (see: A Comparison of Google Web Search and the Darwin Awareness Engine™). 

Darwin allows you to become your own curator for topics of your interest. You can also go beyond this to act as a curator for friend or colleagues You can set attractors on your topics of interest: people, places, concepts, and more. Then you can see what emerges. This can be especially valuable for niche areas. You can create your own online magazine. It can cover a much greater array of source than Google News and it will show the relationships between news items that emerge rather than simply displaying them. You can adjust your filters to further focus you’re your curation efforts. We see content curation as one of the major applications for the Darwin Awareness Engine™.

Comparing Google Web Search and the Darwin Awareness Engine™

I want to share this post I first wrote for the Darwin Awareness Engine blog last week to see what you think and what questions you might have. Google has been the dominant web search engine for years now for good reason. It is the default opening screen on my laptop and I use it multiple times every day. It brought the innovative concept of Page Rank to gauge popularity and provide us with useful, quality content.  The Darwin Awareness Engine™ brings a different approach through its use of Chaos Theory and the concept of correlated content. I am certainly not attempting to equate these innovations but I want to attempt to better explain what the Darwin Awareness Engine does by comparing it to the well-known Google Web Search. First, let’s review what Google does.

You put a search term or phrase into Google. It brings back a list order by its algorithm based on reputation (link popularity), relevance (based on content), and currency.  Google decides what is important for you to see. Lately Google has started to try to better understand what you do to bring more focused and personalized results. However, they are still the “decider” and the list is one dimensional with the recent addition of some social media sidebars. 

This works very well but it can be subject to spammers who attempt to manipulate the dimensions of reputation and relevance through nefarious means.  There has been some concern of the impact on legit content (see ReadWriteWeb on Content Farms: Why Media, Blogs & Google Should Be Worried and Tech Crunch on The End Of Hand Crafted Content). There are also the more honest search engine optimization people (SEO) who are simply trying to get the best returns for their honest content.  For example, I help companies with their blogs and one of the services I offer is SEO for legit content.

 Applying SEO to Google can lead to some interesting results. A Google search on - beach bars Sardinia – finds a blog post on beach bars in Sardinia I did based on a vacation trip as number one out of over 250k results. As far as I know it has held that position for the past four years. Now granted I did original field research with a friend and it is part of a series, Rating the Beaches of Italy, which also remains number one in Google (out of 800k) for that term. While my content is legit, I have to admit that I am far from the best expert on Italian beaches and their bars.  The listing on Google is more the result of using good SEO to get there and the fact that it is a niche topic.

Google is trying to determine the proper order of the universe of content. It does a great job but it is up against an impossible task according to Chaos Theory. It will not achieve perfection and it will have to try to stay ahead of spammers and coexist with legit SEO along the way. Darwin’s Awareness Engine does not attempt this task. It accepts there is no order in the universe of content and let’s you be the decider. It just tries to expose you to the different dimensions and themes in your topics of interest. It tries to make you aware of what is going on so you see trends and find things you were not aware of.

The Darwin Awareness Engine uses the concept of attractors (based on Edward Lornez’s attractor concept – see math model in illustration w via Wikipedai) to let the content order itself through correlations of related   themes that emerge. Instead of a list ordered by Google you get a Scan Cloud™ that displays the content related to your chosen topic that is organized by the themes that have emerged that correlate with your content.  Under each theme is a set of related content and you can drill down to this content. You can decide which themes to explore. You can also decide whether you want to see the results form informal (blogs, etc.) or formal content sources (mainstream news sites, etc.).

If you are interested in climate change you see content organized under such themes as - threats - or - hoax - depending on your view and, of course, see what the other side is saying. You can also see what the English speaking press is saying about the Copenhagen conference or what the French speaking press is sawing by looking under the English or French spelling of the term Copenhagen. 

You can see content emerging in real time that is not order by popularity links and does not have to wait for the links to be created for it to emerge in search results.  This allows for unanticipated results that might be buried in the back pages of a Google search for days, if not forever. For example, when looking at content related to the prime minister of Quebec we found coverage of the current initiatives in electrical service to the Canadian Maritime provinces. This was expected. We also found some unexpected emerging concern by the oil companies over this move as the theme – oil – appeared in the Scan Cloud. This was not expected.

The Darwin Awareness Engine is not designed to replace Google or other search engines. These tools are largely designed to help you find what you know you are looking for. With many search engines you can narrow this focus with filters but these are filters that you select. In contrast, the Darwin Awareness Engine is designed to provide an alternative way of letting you discoverer what is happening in your areas of interested, even the unexpected. Here is more. 

Coveo Expresso Available as Free Basic Level Enterprise Search Tool

I want to share this post I did that first appeared in AppGap to make you aware of a free search tool for those in a Microsoft environment. Coveo offers enterprise search technology including their enterprise search modules and search-based applications for such functions as call centers and litigation & compliance. It has now launched a free, entry-level enterprise search solution, Coveo Expresso™ Beta.  Coveo’s new offering provides a basic, but robust, set of search functionality, at no cost, for up to 50 users, 1M desktop files and email items and 100,000 Intranet documents.   I recently spoke with Louis Tetu, executive chairman, Laurent Simoneau, CEO, and Diane Berry, VP Marketing & Communication, about this new offering.

We first discussed the top two of their three levels of offerings to put Expresso in context.  Louis describes their three tiers. At the top are enterprise search solutions that apply their search technology to specific business processes such as call centers, human resource management, knowledge management, e-commerce, and others.  I am especially interested in their call center solutions, as I have tackled this issue from the knowledge base side.  In these cases having good search often was a critical factor. I can certainly see the logic of making search the front end and leaving the content in place. We will discuss these solutions in more depth in a subsequent conversation.

Their enterprise search modules cover such areas as desktop search, email search, CRM search, intranet search and search through mobile devices. A subset of these modules forms the core for Expresso and they are the migration points for people who want to move beyond what the free product offers.

Next, Laurent took us on a tour of Expresso. We began with the email search. You can search your own email account to find both conversations and attachments. You can also filter by facets such as document type, person, date, etc.  In addition, there is an Outlook Sidebar, a search plug-in, which provides related conversations, related people, and related attachments as a side bar item. You can search any indexed content without leaving Outlook. If these results are not sufficient you can launch advanced search with guided navigation through search facets. Here is a sample email search screen below. You can see the search results in the left column and facet related information in the right column.  You can search other’s email account if they give you permission. 

 

I could see the value of this capability from the start and wanted a copy for myself but they have currently focused Expresso on the Microsoft world which makes a lot of business sense as that is over 85% of the market. It works with Exchange Server, Outlook, Active Directory, Sharepoint, and related products. You can get Coveo’s capability for other platforms including the Mac environment, where I work, and Lotus Notes through their advanced search modules.

 

With the Outlook Sidebar shown below you can select a person and see all the conversations with this person, as well as attachments. They are sorted by date. As I mentioned, if you want to move from browsing these lists, you can still bring up the search box.

 

 

Next, we covered the desktop search that looks at all the content on your computer.  Unlike some search engines this tool does not actually process its searches on your computer, taking up processing power. All of the content on your machine is pushed to a server and searched at the server level. You can exclude content from this process for privacy and/or security purposes.  Security can be at the file level so you have granular control over what occurs.  Below is a sample file search.

You can also search your company’s intranet.  These searches can be ordered by relevance or date. The relevance factor is available on most search types. The relevance of a document is determined by a mix of 75 factors. Your administrator can set weights on these factors such as credibility of author, popularity of file, etc. A sample set of intranet results is shown below. Again you see the results in the left column and facet related information in the right column such as document type and author.  The type of facets offered varies by what type of search you are conducting.

We also looked at a people focused search. You can pull information from Outlook, Exchange Server, and Active Directory. If you are using Salesforce.com you can also pull this content into the search process. You can even narrow the search to a subset of your Salesforce.com data such as a particular pipeline, though not in the free version, currently.

There is an administrator dashboard that shows you how much of the free capacity you have utilized (e.g., users, files, etc.) and it conveys how to get more capacity if needed.  I think this free offering is an excellent sales tactic. Since you access this product in a self-service mode, the cost of sales is virtually zero. 

There is only upside to getting it in the hands of people who will likely want the additional features and capabilities available at the next tier. At the same time there is enough functionality at the basic level for users to see the value. These users are also not taking a risk as there is no time limit to the free offer so they can continue to make use of the tool and their investment in understanding it even if they do not move to the next level.  I am impressed with what they are doing. Others seem to feel the same way as they recently received $8.2 million in Series B funding led by BDC Venture Capital.

Coveo is based in Quebec City, one of my favorite places in this world. I have been there many times as it is driving distance of Boston. Louis very kindly gave me an excellent collection of great local restaurants to add to my list. I will start trying them the next time I can get up there.

 

 

Text Analytics Offers a Discovery Channel to Make Better Sense of Content

Socrates said that, “the unexamined life is not worth living.”  In perhaps a parody of this statement the Forrester report, Text Analytics Takes Business Insight To New Depths, states that “unexamined content is a wasted opportunity.” They look at an array of tools that go beyond simple search. This might be a good thing as a Google search on the term, the unexamined life is not worth living, turns up a number of term paper services in its top ten.  The Forrester team of Leslie Owens with Matthew Brown, Sara Burnes, and Peter Schmidt conclude through the report subtitle that “An Obscure Technology Has Found Its Killer App.” I would certainly agree and appreciate receiving a review copy of this report. 

The authors note that customers, employees, and competitors comment on products, personalities, and companies in increasingly public places, like Twitter and discussion forums. Forrester that online ratings and reviews topped the list of trusted material. The abundance of news and commentary on the Internet could contain actionable knowledge for your business. This is not simply a Web issue as internal sources like emails and call center notes are full of product suggestions, feedback on competitors, and thoughts on the market.  With the rise of enterprise 2.0 and related social media data, the opportunities for data mining and information overload are expanding. To make sense of all this content, businesses are turning to text analytics tools.

The team concludes that this “little-known technology has a compelling value proposition: extract meaning out of large quantities of text by mining, interpreting, and structuring information to reveal hidden patterns and relationships.” I am not so sure about the little known part but I agree with the rest of the statement.  Perhaps I am biased because I have been talking with a lot of vendors in this space.  One of the great opportunities of both Web 2.0 and enterprise 2.0 is all the conversational content that it creates. You would be losing a good bit of the value it you do not look at this content but you need good tools to do it.

The team found that here is too much content to review by hand, It is hard to separate a signal from noise, and here is no way to know what questions to ask.  They add that when you don’t know what’s significant about a set of content, you can’t investigate it with a search engine. The report reviews a number of the major text analytic players and should be useful for those making decisions in this space. Text analytics software typically includes four key components: entity extraction, categorization, relationship mapping, and sentiment analysis.

The authors caution that “until recently, text analytics software was obscure and academic, used primarily by early adopters in the life sciences field… Text analytics is a framework and a process. It is not always necessary or possible to deploy text analytics as a single, linear solution from a single vendor.” It does seem that many of the solutions are complex and are designed to be used by professionals, often with subject matter expertise in linguistics and statistics. There is much more and you can obtain the report at the Forrerster web site.

As a post script to my comments on this useful report I want to say a bit about Darwin Ecosystem. Readers of this blog will know that I am connected with Darwin. The Darwin Awareness Engine™ offers a different approach to the same issue. Using chaos theory it uncovers the emerging themes about a topic, a form of categorization. However, it operates through a simple interface that is designed to be accessible by a wide range of people. The Scan Cloud™ visualization allows you to immediately see the relationships between content groupings in real time within your chosen theme and draw your own conclusions as to what trends are occurring.  It is another way to get into content discovery. Putting The Darwin Awareness Engine in the hands of experts, will better equip these individuals to discover new trends in their field. However, ii is also accessible to the average business user so they can have a greater awareness of their areas of interest.