Portals and KM

Further Reflections on Dave Snowden’s Resilient Organization

Last November I attended the KM World 2010 the keynote session, The Resilient Organization led by Dave Snowden, Founder & CTO - Cognitive Edge. Here are my session notes. There were a number of things that Dave said that strike at some of the principles behind the Darwin Awareness Engine™. I wanted to reflect on those touch points in this post as we enter the new year.

First - Dave noted that most people can produce a decent meal through recipes and the right ingredients. A master chef can produce a great meal with whatever is your kitchen.  We need theory and not just recipes to blindly follow.

We developed the Darwin interface to be very simple to use. It only takes a few minutes to understand how to make use of the data visualizations. However, it works best under the hands of a master who really knows the content, can select the right feeds and make hypotheses to test through following feeds, Rob Paterson’s work with KETC is a great example (see KETC TV Social Media Experience Part Four: How Does Darwin Help Clarify Complex Issues?)

Second – Dave noted that the human brain scans a small percentage of the available data and makes quick decisions. The only people who look at all the data are autistic and they are overwhelmed but management theory adopts this autistic approach.  To react to the charging lion you cannot examine all the data before you react.

We designed both the Scan Cloud™ and the Buzz Tape™ for quick scanning (see below) only you to check out a vast amount of data quickly and rill down where you find interesting things. Anomalies show up.

Third - Dave said that despite what Chomsky said, there are no deep structures in language and so taxonomies are useful but limited.  We would certainly agree at Darwin since we let the content self-organize dynamically rather than impose an external structure. We are the opposite of semantic technology. If you let the content self organize there is no external framework like Google Page Rank to game or approach through legitimate methods. There is no SEO or even spam as it would serve no value. We allow the visualization of connections between content related to a particular theme to reveal new relationships and help with the discovery and awareness of trends, both anticipated and unanticipated.

Rather than taking a linear, deterministic approach to finding content through seeking repeatable patterns on the Web with an external framework like page rank as done by Google, we use Correlation Rank™ to find non-repeatable patterns in Web content that come and go in real time. Rather than using an external framework, we look for self-organizing patterns within the content itself.

Fourth - Dave noted that technology needs to be seen as cognitive augmentation.  Larry Prusack said intuition is just compressed experience. The human brain blends fragments in the face of challenge. He noted how General Patton did this in World War Two. We use provide the bits of content so the human mind can decide on how to put them together. Darwin Correlation Rank™ does not offer a precise answer, but it reveals a movement or trend as it occurs

Fifth - Dave stated that we need to move from aggregation for people to allowing people to synthesize fragments. We need to allow for contextual blending of fragments. We need to put decision makers in direct contact with the raw data. This is our goal at Darwin. 

You can download a podcast and slides of Dave's KM World talk at the Cognitive Edge site.  

 

Our Enterprise 2.0 Boston 2011 Submission: Enterprise Awareness and Weak Signal Sensing

Thierry Hubert and I have submitted a proposal for a session in the Boston Enterprise 2.0 2011 conference based on the ideas in this post. Here is the link to our submission. If you like what we are saying here please take and look and comment and vote for us. Thanks. 

Weak Signal Sensing is the ability to detect anomalies and other valuable bits within complex arrays of information. It is a skill developed over time and is one of the traits that separates deep experts from novices. The use of social media within the enterprise and on the Web has created a rich source of content to mine for insights. It is a data rich environment for Weak Signal Sensing. Recognizing this potential a large number of tools have emerged to try to make sense of this data through such means as sentiment analysis, popularity ranking, and semantic technology. They have taken on the difficult task of trying to get a computer to understand and pick out the valuable bits within the rich corpus of content generated as a byproduct of the transparency to interactions within enterprise 2.0 and on the Web.

At Darwin Ecosystem we take an alternative approach to enabling Weak Signal Sensing with the Awareness Engine™. Rather than have the computer attempt to make sense of content and try the even more difficult task of where to look next, an Awareness Engine allows the content to self-organize into simple visualizations that enable people to better perform these tasks of exploration and discovery.  It avoids predetermined categories that can inhibit discovery of the unexpected. Using such visualizations as a Buzz Tape™ which highlights rising topics of interest and a Scan Cloud™ which allows a quick exploration of the themes within selected content, an awareness engine empowers experts to more efficiently make use of their capabilities. It can also spread Weak Signal Sensing capabilities further within an organization.

Organizations making use of enterprise 2.0 technologies are capturing information that emerges from collaborative discussion, deliberations and ideas that are now stored in systems.  The advent of enterprise 2.0 portals have allowed for the context of teams and projects to be connected for easy content creation and retrieval.  As such, we are previewed to the processes that lead to actions.  Although incredibly useful, organizations often limit the transparency by imposing hierarchical and role-base access to this body of information.  This limitation prevents knowledge workers from increasing their Weak Signal Sensing capability.  Weak Signal Sensing is encouraged and accepted at the highest levels of management as it is acquired from experience and knowledge captured across many different and unexpected sources.  This is what allows a senior executive to deliver credible and strategic value to the organization, whereas junior managers would have to prove their reasoning before being enacted.

The information that can empower ubiquitous Weak Signal Sensing is found in all sources, but it is in one's memory and experiences above all.  This is why you can't automate Weak Signal Sensing; it is personal and the product of one's cognitive ability to sort meaning from noise.  The value of increasing Weak Signal Sensing is paramount in today's changing world and economic models.  An organization capable of improving this faculty will have more actionable ideas and a competitive advantage over stove-pipe and highly process driven companies.

The key to elevating Weak Signal Sensing for all is to provide tools that go beyond search and related records.  Tools that improve awareness and serendipitous discoveries.  Tools that reveal different answers and intrigues according to different users.  Basically tools that awaken one's own memories and experiences to identify actions and information that needs to be shared.

The value of increasing weak signal sensing is paramount in today's changing world and economic models.  An organization capable of improving this faculty will have more actionable ideas and a competitive advantage.

 

 

Analytics Made Easier for IT and the Analysts

Meta S. Brown recently discussed Why IT Doesn’t “Get” Analytics (and Why the Time is Right for Change). Meta noted that in a 2009 IBM survey, more than 2,500 chief information officers from around the world were asked, “What kind of visionary plans do you have for increased competitiveness?” In response, eighty-three percent of these CIOs identified analytics as a priority. Meta concluded that “IT leadership is beginning to understand that it is time to make nice with analysts.”

This should be good news to data analysts as Meta notes that they often have extremely poor working relationships with IT. “Indeed, if you haven’t encountered analysts making end runs around IT to obtain data, subverting software procurement processes and leaving work poorly documented, you have led a sheltered life…At the same time, neither IT staff nor leadership has adequate understanding of the value or workings of statistics, data mining or other analytics.”

At Darwin Ecosystems we have seen a bit of this issue and have designed the Darwin Awareness Engine™ as a cloud application that requires little, if any, support form enterprise IT.  So analysts you can become independent of IT support and track what is happening o in your areas of interest. Even looking inside the enterprise requires relevantly little active IT support but most likely still involves governance issues. We hope that this should be good news for both analysts and IT. 

 

Results for Using Search Queries for Future Predictions are Mixed

The MIT Technology Review recently reported on Future Gazing with Search Data in an interesting article by Katharine Gammon. She notes that computer scientists have praised Web-search data as a way to spot emerging trends. However, recent research shows that the results are mixed and context dependent.

For example, Erik Brynjolfsson, director of the MIT Center for Digital Business, says that Web search queries provide the most accurate predictions in cases where people do research before they make a purchase. His research has shown that a rise in home sales can be predicted from Web search queries.

On the other hand, researchers at Yahoo analyzed search queries and found that such searches are not always the best way to spot a trend. They looked at the volume of search queries related to a particular movie, song, and video game up to six weeks before each came out. However, the traditional critics were just as good in most cases.

It is hard to determine intent from searches and then there might be spammers in operation that compromise the data. The simple volume of searches is a fairly rough measure. The Yahoo researchers said that search data may be useful when a small improvement in prediction accuracy could have a big impact--for example, in the financial world.

The Darwin Awareness Engine™ takes a different approach to looking for trends that can be complementary to these search results.  Instead of looking at the volumes of searches, it looks at the volume of conversations about a topic.  The BuzzTape™ reports topics that are receiving rising attention in real time. The Scan Clloud™ looks at relationships between sub-themes and the main topics of interest.  Since the content self-organizes itself, it is not possible to compromise the results through SEO or spam.

 

 

Lynda Moulton Lists Five Products to Reduce Information Overload

Lynda Moulton is a consultant, an analyst with the Gilbane group, and runs the Boston KM Forum. I have attended it over the past several years. About a year ago we presented a prototype of the Darwin Awareness Engine there.  Lynda recently provided a useful post on: Focused on Unifying Content to Reduce Information Overload.  She noticed while attending the Enterprise Search Summit and KMWorld 2010 in Washington, DC last month the diversity of ways organizations are getting answers to stakeholders more quickly. I have had a chance to review her recent Gilbane Beacon report Establishing a Successful Enterprise Search Program: Five Best Practices. Here is my blog post on it (see Semantic Technologies Explained and Examined). 

Lynda noted five software products she saw at Enterprise Search Summit and KMWorld 2010. She said that, “these products, in alphabetic order, are all notable for their unique solutions tailored to different audiences of users and business requirements. All embody an approach to unifying enterprise content for a particular business function.”  The products were: Access Innovations, Attivio, Coveo, Darwin Awareness Engine, and RAMP.  I have written about Attivio and Coveo before and have great respect for them. We were very pleased to be included. Here is what Lynda said about Darwin:

“Darwin Awareness Engine was named the KMWorld Promise Award Winner for 2010. Since their founder is local to our home-base and a frequent participant in the Boston KM Forum (KMF) meetings, we are pretty happy for their official arrival on the scene and the recognition. It was just a year ago that they presented the prototype at the KMF. Our members were excited to see the tool exposing layers of news feeds to hone in on topics of interest to see what was aggregated and connected in really "real-time." Darwin content presentation is unique in that the display reveals relationships and patterns among topics in the Web 2.0 sphere that are suddenly apparent due to their visual connections in the display architecture. The public views are only an example of what a very large enterprise might reveal about its own internal communications through social tools within the organization.”

Thanks to Lynda for these comments and her post. 

 

My KM World 2010 and Enterprise Search Summit 2010 Notes: Facilitating Content Discovery Using Chaos Theory

Here is another in a series of session notes for KM World 2010 and Enterprise Search Summit 2010. In this case it is the session I did - Facilitating Content Discovery Using Chaos Theory. Here is the session description. My notes follow. 

“The presentation provides a practical application of chaos theory for creating useful content focal points from the chaotic mess of content resources distributed across the web. Several examples will be provided, both from the internet, where socially generated content is exploding, and from the enterprise, where social tools are contributing to content bloat. The examples illustrate how activity can make information more discoverable and easier to explore through various visualization techniques.”

To set the stage let’s go back to the creation of the text we use to capture content. One of the greatest information technology inventions remains the phonetic alphabet. The Greeks came up with it in the early 8th century BC. The same twenty-four letters have been used to write the Greek language ever since. It is the first and oldest alphabet to note each vowel and consonant with a separate symbol.  This invention provided great flexibility and unprecedented accuracy of linking symbol to word and thus on to thought.  Now we moved from relying on informal communication through spoken conversations to formal documents enabling many breakthroughs.

One of the great flexibilities of the phonetic alphabet or text was the wide range of vehicles that can be used to convey text. Most of the technological advances since the work of the early Greeks have been on the vehicles to convey text, such as paper over clay or stone, and the means to produce these vehicles, such as the printing press, rather than the alphabet, itself.  Text remains king on the Web and the darling of search engines.

In the past, the effort to create text acted as a filter but a series of inventions such as the printing press cut into this difficulty reducing the barriers to creating formal context, generally for the good of most.

Now with Web 2.0 the barriers to creating permanent content through text have been mostly shattered. You still need a computer or a mobile phone or some other smart device and access to the Web.  But much of what is being added is closer to the informal conversations of the pre-text world, than the type of content normally taken to print.

The economist Charles Handy noted that with the knowledge economy that knowledge workers now own the means of production, their minds, unlike their industrial age counter parts. Tings will change. Well with content production the speakers and readers how have the means of production through the Web. This has both good and bad consequences. As Plato said in reaction to a possible over-reliance on documents, conversations can clarify intensions. On the plus side we have much more to look at. One the negative side we have much more to look at.

In 2008 more content was created through the Web than in the history of content. The same thing happened in 2009.  We are now in danger of being overwhelmed by content and sinking into chaos.

The first Web search engine was created in 1990 by computer science students at McGill University in Montreal.  It's searchable web site database was limited as it did not index the content.  It is in 1994 with WebCrawler and Lycos that full-text search appeared on the WWW.  

Google added a new and transformative dimension to search with it s page rank concept. Let the readers help determine what content has the highest value and also look at currency, along with relevance   This has become very useful. I use it every day and it is the default home page for my laptop. I primarily use Google when I know what I am looking for. For example, when I want to find the score of Wizards Celtics NBA game last night or the Web Site of the Boston Celtics or an article I heard about at this conference, I turn to Google. 

However, Google Page Rank was created before the explosion of Web content and has its limits. Sometimes those better at speaking Google’s language than the others on the topic are elevated above the real experts. This is a problem when you use an external framework. People learn how to game it for good and bad reasons. SEO and spam are unintended byproducts of Google.

We do not see Chaos Theory and Correlation Rank™ as a replacement for Google. We see it as a complement. It is an awareness or discovery engine not a search engine. Instead of looking for repeatable patterns, its looks for correlations between content to discover new relationships beyond what you where initially looking for.

If you use Correlation Rank to let the content self organize there is no external framework to game or approach through legitimate methods. There is no SEO or even spam as it would serve no value. 

If I want to find the stories I did not know to look for about a topic of interest, if Ito discover new things, if I want to see the breaking news in real time, the stories generating the most buzz, I turn to Correlation Rank. It does not offer a precise answer, but it reveals a movement or trend as it occurs.  I use Correlation Rank when I want to explore any area of interest in more depth and when I want to see real time alerts.

How is it visualized? Now if I go to Google I get a linear list of results rank ordered by its external framework. Google decides what is important for you. Through the Scan Cloud™ the Darwin Awareness Engine™ provides an overview of the 100 top themes related to your topic of interest visualized in a tag cloud type structure and allows you to explore the ones that appeal to you (see below). You become the decider. In this way we see it as very similar with many of the ideas that Dave Snowden discussed in his session that I posted on yesterday. 

Through the Buzz Tape™ you see topics of rising interest.  Then you can make one of these rising topics the focus of a new Scan Cloud.

 

 

At KM World 2010 Today: The Darwin Awareness Engine™ Receives 2010 KM Promise Award

I am very pleased to announce that the Darwin Awareness Engine™ won the 2010 KM Promise Award today at KM World.  It is a great honor. Here is the description of the award from the KM World site.

“Many companies promise that their technology is the best knowledge management solution. One of the greatest challenges for organizations purchasing these technologies is to determine which of these companies will deliver on this promise. This award is given to the organization that is delivering this promise to its customers by providing innovative technology solutions for implementing and integrating knowledge management practices into their business processes. The award-winning organization demonstrates how it goes beyond simply delivering technology to working with clients to ensure that both the technology and knowledge processes are imbedded into the work processes. In other words, it helps organizations realize positive business results.”

We are in great company. Last year’s winner was Inmagic and the 2008 winner was Attivio.  Both are great companies that I have covered several times (for most recent see Inmagic Presto 3.5 Broadens Its Collaboration Capabilities and Attivio Aligns with Traction and Releases New Features). Here is a conversation I had this year with Probstein, Attivio CTO.

Here is a video of the award presentation made by Kurt Kragh Serenson of IntraTeam. 

 

KM World is a great event that is currently in its 14ht year. I remember going to the first three and many since then. I am now covering a number of this year’s sessions on this blog.  Here is a complete listing of my KM World 2009 Session Notes.

We have recently released a series of themed Darwin Editions™ powered by the Awareness Engine™ that focus on specific topics to better demonstrate its capabilities and as a service to our readers. These can each be accessed at no cost. You only have to go through a brief registration process and we promise you that no sales person will contact you as a result.  

Here are links to a series of posts going into more detail on the first three Darwin Editions and how to use them:

Darwin Awareness Engine™ Edition Series: Part One - Explanation and Instructions

Darwin Awareness Engine™ Edition Series: General News

Darwin Awareness Engine™ Edition Series: Social Media on the Web

Darwin Awareness Engine™ Edition Series: Social Media in the Enterprise

Darwin Awareness Engine™ Edition Series: Energy Sector News

We hope that you find these Darwin Editions useful and will certainly appreciate any feedback. Also look at Twortex, our business partner which offers a Twitter search tool using the Darwin Awareness Engine. Please contact us at [email protected]. We will also be happy to discussion the various business options for a fully functional Darwin edition with all administration rights.  You can create your own Darwin Editions on the topics of your choice with the target content of your selection.  There are both individual and enterprise options. 

 

 

Initial Darwin Edition™ Series is Now Available

We have decided to provide a series of themed Darwin Editions™ powered by the Awareness Engine™ that focus on specific topics to better demonstrate its capabilities and as a service to our readers. These can each be accessed at no cost. You only have to go through a brief registration process and we promise you that no sales person will contact you as a result. 

The initial themes for our Darwin Editions include: general news (our original alpha), social media on the web, social media in the enterprise, US immigration news and discussions, and oil and gas industry news. The last two will be released over the next few weeks and the first three were made available last week (see links below). In this context we also point to our partner, Twortex, which uses aspects of the Darwin Awareness Engine to provide a free Twitter search tool.

 
Here are links to a series of posts going into more detail on the first three Darwin Editions and how to use them:

Darwin Awareness Engine™ Edition Series: Part One – Explanation and Instructions

Darwin Awareness Engine™ Edition Series: General News (500 sources)

Darwin Awareness Engine™ Edition Series: Social Media on the Web (over 160 sources)

Darwin Awareness Engine™ Edition Series: Social Media in the Enterprise (over 170 sources)

Darwin Awareness Engine™ Edition Series: Energy Sector News (over 100 resources)

These Darwin Editions provide a way to monitor the conversations within several focused areas. They offer many of the user capabilities that come with the complete Darwin Awareness Engine. Darwin looks at targeted content rather than spidering the whole Web. This targeted content approach provides greater control to select relevant content and can be focused within the Web or within the enterprise through such applications as SharePoint, Domino, and many of the other collaboration and content management platforms.  Determining and curating the target content is one of the administration function that come with the commercial versions. In this Darwin Edition series we have picked the content sources and will continue to refine them and add to the total sources.

The default view shows you what has happened in the target content sources in the last hour. You can expand it to the last two hours, 24 hours and even last 200 hours. You can also create "attractors" which are queries that allow you to look at how specific topics are being addressed within the target content. There are other options and you can find the features described in our instructions post. The color coding in the Scan Cloud represents different types of terms such as people, events, and organizations. They will be explained in the instructions and are an admin function. 

We also used the Traackr A-List for PR 2.0 for some of the social media in the web sources and a Traackr A-List for enterprise social media that we developed for some of the choices in that Darwin Edition. These lists of 25 top influencers use a group of keywords you define and update weekly so we continue to monitor them for possible new sources. 

We hope that you find these Darwin Editions useful and will certainly appreciate any feedback. Contact us at [email protected]. We will also be happy to discussion the various business options for a fully functional Darwin edition with all administration rights.  You can create your own Darwin Editions on the topics of your choice with the target content of your selection.  There are both individual and enterprise options.

 

 

Darwin Awareness Engine™ Covered in eContent Magazine

I have great respect for eContent Magazine and was very pleased to see the following coverage for the Darwin Awareness Engine™

“Darwin Ecosystem released the Darwin Awareness Engine, a product able to analyze complex underlying patterns in markets over time in addition to the events that have an impact on these markets. The Awareness Engine further allows organizations to create, test, and refine hypotheses regarding the driving forces behind market events, and compare and inform those hypotheses with conversations and data that appear on the web. This web data is presented in a Scan Cloud, which uses correlation between key words as its basis, as opposed to the relevance ranking of search engines. Darwin reports that the Darwin Awareness Engine disposes of the need for organizations to search for sources manually, which can be a time-consuming process.”

Michelle Manafy is the Editor of EContent magazine and the Intranets newsletter. Michelle is also Chair of the Enterprise Search Summits and Buying & Selling eContent Conferences. The call for Speakers for next May's Enterprise Search Summit is out. I went this year and found it to be a great conference. It was also our first conference to present the Darwin Awareness Engine. Here is a sample of notes from the May 2010 event. 

 

The Darwin Trend Analysis Offering

Here is a post about a new service offering at Darwin Ecosystems. We now live in a complex and inter-connected world where reliance on simple explanations based on surface analysis no longer allows an organization to remain competitive.  Whether you are the CEO of a large global company or the head of a small local organization, what happens in the world as reflected through conversations and events on the Web affects your ability to survive and prosper.

In addition to providing real time discovery of emerging trends and issues, the Darwin Awareness Engine™ has an ability to support the comprehensive analysis over time of complex underlying patterns in markets and the events that affect these markets. You can create, test, and refine hypotheses about root causes against the conversations and events that are transpiring on the Web and through it.  You can look at what shapes your markets and topics of interest to make better predictions and discovery new ways to solve the challenges you face.

For example, our client KETC, the St. Louis public television station, was able to test and refine root causes on immigration issues by setting up related feeds and topics with the Darwin Awareness Engine™. They were able to undercover trends that might have been difficult or impossible to discover through traditional search and then feed these discoveries into their community for further exploration. Darwin is now being used to monitor the impact of these discoveries on the ongoing conversations within their community. For more detail on the KETC social media efforts see our five part series: The KETC Social Media Experience. 

You can use the Darwin Awareness Engine™ to a tackle the complex issues your organization faces or we can provide the expertise to perform the analysis over time for you.  Here is how it works.

Step One: A Darwin expert will work with you to develop testable hypotheses about the issues and challenges that you face.

Step Two: Appropriate content feeds are selected through a number of methods and the initial attractors (topics of interest) are determined.

Step Three: The results from the initial feeds and attractors are examined on a daily basis and both the feeds and attractors are refined.  Your initial hypotheses are also tested and refined over time. This is usually requires a few weeks.

Step Four: The refined hypotheses are further tested and shared with appropriate members of your organization and/or community for their reaction and the development of actions plans that align with the results.

Step Five: The actions plans are put in place and Darwin is used to determine the effectiveness of these actions and to provide input for their ongoing refinement. 

Step Six: We can train you to carry on this type of analysis, action, and refinement for existing and new issues over time or we can continue to work together to support your efforts.

 

 

Darwin Awareness Engine™ Named a 2010 KM World Promise Award Finalist

I am very pleased to mention that the Darwin Awareness Engine™ was named a finalist for the KM promise award sponsored by KM World.  The winner will be announced in the November/December Issue of KMWorld with distribution at the 2010 KMWorld and Enterprise Search Summit in Washington DC which will occur November 16-18.  I will also be speaking on Facilitating Content Discovery Using Chaos Theory as part of the Knowledge Showcase. My session is from 12:30 to 12:45 on Thursday, November 18.

Here is the description of the award from the KM World site.

“Many companies promise that their technology is the best knowledge management solution. One of the greatest challenges for organizations purchasing these technologies is to determine which of these companies will deliver on this promise. This award is given to the organization that is delivering this promise to its customers by providing innovative technology solutions for implementing and integrating knowledge management practices into their business processes. The award-winning organization demonstrates how it goes beyond simply delivering technology to working with clients to ensure that both the technology and knowledge processes are imbedded into the work processes. In other words, it helps organizations realize positive business results.”

We are very honored to be named as a finalist for this award.

 

Interconnected Business Trends from Michael Fauscette

I have been following Michael Fauscette on Twitter for some time and enjoy his updates, many include links to helpful content. Last month he posted a comprehensive set of predicts, Evolution of Change: Signs for the Future of Business, that tie different trends together.  Michael draws on "The Singularity is Near" by Ray Kurzweil and the concept that we often underestimate the rate of change but and tend to look at change in one area in isolation rather than as a part of a set of related changes. Michael links some current change factors to conclude there are some dramatic and fundamental changes ahead for business. I would agree and a part of this, but certainly not all, is related to the combined set of initiatives grouped under the term enterprise 2.0.

I will not review all of the trends and I encourage you to look at Michael’s complete post. A number of them relate to what we are doing at Darwin Ecosystem and these are the trends I will focus on here. They include: cloud computing, everything as a service, networked business (customers, employees, partners), the power of community, internet of things, big data, predictive analytics, multifaceted business models, and necessity driven innovation.

Like many other firms we chose to offer a cloud solution and a services approach for the flexibility we can offer and the simplicity that our clients face in terms of technical implementation. We can host the Darwin Awareness Engine™ for a client or clients can do this themselves. In both cases there is little effort to integrate with their current technical architecture and/or involve the IT guys, even when Darwin is targeted at internal enterprise content in sources like Sharepoint and Domino, or multiple subscription services as Forrester, Gartner, etc.

The Darwin Awareness Engine™ can aggregate content across the networks that a business sets up or you can look within individual components of these networks. Clients can see the emerging themes and conversations their employees and business partners are holding. Michael writes, “The Internet creates hyper-connectivity and opened up the flow of information around barriers. The social web changed interaction models and redefined community.” Now you can look within these communities to better understand emerging themes.

The “internet of things” extends to television and opens up new models for traditional news media. Now you can combine traditional media and social media. Darwin lets you understand the emerging themes and breaking news. One of our clients, KETC, the PBS affiliate in St. Louis, is using Darwin as part of a pilot program connecting social media and television. Through Darwin they look for breaking news and trends within their news targets and then plan to also make use of Darwin to better understand the impacts of their new efforts within the program.  

The proliferation of data and the need to look with this data for predictive analytics is the foundation on which the Darwin Awareness Engine™ was created.  In 2008 more digital content was created than in the history of content and then the same thing happen again in 2009. Rather than try to impose external frameworks or use semantic techniques to try to comprehend this exploding content, we use Chaos Theory to allow the content to self-organize. Then we use data visualization approaches to allow the human user to determine which themes are in important for further exploration.  In the process we avoid the potential for spam and the practice of SEO that other approaches bring.

Finally, we are exploring multifaceted business models, enabled, in part, by our cloud services approach. We offer can Darwin’s capabilities as a direct service, through other software firms to enhance their content discovery process, through content focused business partners that publish their own niche publications, and as niche solutions that set on top of the core technology. We began as a necessity driven innovation because of the difficulty imposed by current models of content discovery that often required complex semantic techniques and/or the need for taxonomies.  This led us to Chaos Theory and the use of a radically different approach.

Thanks again to Michael for bringing these trends together and helping us reflect on their interconnections. 

NPR is Expanding its Traditional Media Audience Through Social Media

Rob Paterson recently posted on: It's not the content but the connection that counts - NPR Rocks. He said that NPR is trying to attract an younger audience and has succeeded “not because of a content shift but because they made it easier for a younger audience to connect to content on their terms! The secret was in the flexibility of the new connection NOT the content.” They used social media and mobile apps to achieve this connection.

Rob points to Jolie O’Dell’s post on Mashable, How NPR Is Leveraging the Twitter Generation. She made this statement that I totally agree with.  “…the future of news media lies in successful integration of social media to get the attention (and click-throughs) of a younger generation — a generation whose news needs are vastly different than those of the generations that preceded it.” We recently cover how our client KETC, the PBS television station in St. Louis, is engaging in a successful integration of social media,

NPR is becoming a poster child for this social media effort. They found In a recent survey of more than 10,000 respondents, that its Twitter followers are younger, more connected to the social web, and more likely to access content through digital platforms such as NPR’s website, podcasts, mobile apps and Facebook. The survey found that, 59% of respondents said they use NPR.org, 39% listen to NPR’s podcasts, around half use an NPR mobile app and 28% access NPR via Facebook. The survey also found that, 77% of NPR’s Twitter followers said they get all or most of their news online. These Twitter followers are more likely to expect breaking news, likely because of Twitter’s real-time nature.

The Darwin Awareness Engine™ works in real time and can play a vital role in these social media integration efforts as it is doing at KETC.  In addition, the Darwin powered Twortex Twitter content discovery tool can help you find out what is happening on Twitter. 

Twortex Provides Twitter Search through Darwin Ecosystem’s Awareness Engine™

It can be addictive. Twortex is a Montreal based company that offers a free Twitter search engine powered by the Darwin Awareness Engine™.  I now use it to discover new conversations on Twitter and sometimes find items related to my topics of interest that I did not anticipate were connected to these topics. The Darwin Scan Cloud™ provides the top hundred themes connected to an attractor, our term for a search topic of interest. As you scan over it you can see related themes. Here is a Twortex Scan Cloud™ covering the term – enterprise20 – for the past 24 hours on 9/10. The actual tweets connected to these terms are seen on the right of the screen. Clicking on the listed tweet can take you to the actual tweet.

 Here is the search on tweets for the past 120 hours on 9/10 on the same term – enterprise20 – for the past 120 hours on 9/10. You can see the changes on coverage.

In the screen below you can see some interrelated terms highlighted in red after mousing over one of the terms in the scan cloud shown above.  This provides an alternative to traditional Twitter search that presents topics in a liner order that can make it harder to see trends and more difficult to quickly scan over a large amount of content. You can go to the Twortex site and register and to use it at no cost.  

 
 

Traditional Media’s Move Away from Community has Opened Doors for Social Media

Mike Frey through Social Media Today provided an insightful post on How Traditional Media Created Social Media. Mike opens with this observation. “Prior to conglomeration there was a time that radio, television and print were local community outlets.  At one time the big three were locally owned, and concentrated on local community.  Now most decisions that are made for communities regarding their major media are made from thousands of miles away.  What music to play, what programming to air, and all the cuts made at the print level most stories come off the wore than actually written by reporters.”

I also remember this local control in the three traditional media and have been disappointed in its passing.  Now many people use social media to make up for this loss of community as Mike notes.  I certainly get much of my news through my Twitter friends and share back what I learn through Twitter and blogs. Like Mike, I stopped my subscription to our Boston print paper as it got smaller and, more importantly wrote about stores I had already heard through social media. I found myself often failing to walk down the driveway to pick it up. Now I still read the New York Times online for national and global stories. Even the TV news is a bit old and I watch more to see how they spin stories than to get new news.

Some traditional media outlets understand this trend and are looking for ways to become more connected to their community through social media. I have written about many of these efforts on the Darwin blog (for example, Foursquare Forging More Partnerships with Traditional Media and BBC Pushes Social Media for Its News Staff for Both News Collection and Impact Monitoring).

The Darwin Awareness Engine™ can help traditional media with both news collection and impact monitoring. For example, it can look at social media to bring forward stories for traditional media (see Tracking Social Media Buzz with Darwin Awareness Engine™). Here is the first of a three part series on how Darwin can help make traditional media better connected with its community: Creating a Sustainable Ecosystem for Community News Media - Part One: Providing Relevance and Creating Loyalty. 

KM World Selects Darwin Ecosystem as a Trend-Setting Product for 2010

 In this September issue of KM World, Darwin Ecosystem’s Awareness Engine was selected a trend-setting product of 2010. Found on page 19 of the print version, Darwin Ecosystem (darwineco.com) is defined as: Awareness Engine - an enterprise solution that fetches and correlates its own Web 2.0 information sources, as well as WWW selected sources, to provide timely awareness about what is happening in the enterprise and the internet.  

We were fortunate to appear twice in this issue. There is also a front page article: Breakthrough from Darwin Ecosystem. It notes that Darwin “ applies chaos theory principles to reveal emerging patterns and correlations that accelerates the user’s understanding of what is happening about a given theme of interest regardless of popularity ranking.”  The Darwin Awareness Engine™ provides “a new and missing perspective. The Darwin interface provides an intuitive, interactive ScanClod (patent pending) that reveals correlated themes for contextual content filtering around the user’s topics of interest, a BuzzTape that displays themes that have high content acceleration within the past 24 hours, and the ability to save attractors(a term used in chaos theory) to monitor the evolution of favorite topics.”

We really appreciate this coverage. Thanks to KM World.

 

How Will Search and Social Media Converge?

I read an interesting interview recently. Steve Rubel is senior VP-director of insights at Edelman Digital discussed the convergence of search and social media. He makes a nice distinction between the two. “Search is an intent-driven medium, where users seek out what they want. Social networking is where the content finds you through the lens of friends. Those two are separate, but I think we'll see a real convergence here where search will get a lot more social and social will get a lot more searchy.”

You can now search your friend’s content. For example, with Google you can bring in the results from your social connections via the Google address book. At the same time you can go into Facebook and search your friends' content.  As a result Steve predicts that push and pull will merge. We will become more media-agnostic through an aggregator like Google News, along with a social stream, where we go in and out.

I like this prediction as the Darwin Awareness Engine (TM) can serve the same aggregation function as Google News but with more robust features. The Awareness Engine allows you to spot trends through a combination of content, including your friends’ content or their content choices.  It will be interesting to see where this goes. 

Darwin Awareness Engine™ Covered by KM World Magazine

 I was very pleased to see the coverage of the Darwin Awareness Engine™ in KM World Magazine in their breaking news section under a piece titled, Real-time awareness. It said that “Darwin Ecosystem believes this organic and self-organizing model is best adapted to the evolution of today’s Web when compared with the increasing limitation of page ranking offered by search engines. Although not a replacement for search engines, Darwin’s Awareness Engine highlights a new and missing perspective.”

It also mentioned that “The Darwin interface provides an intuitive, interactive ScanCloud (patent pending) that reveals correlated themes for contextual content filtering around users’ topics of interest, a BuzzTape that displays themes that have high content acceleration within the last 24 hours, and the ability to save attractors (a term used in chaos theory) to monitor the evolution of favorite topics. Darwin's Awareness Engine can be viewed in action here.”

There is more and we encourage you to go to the enterprise article. This coverage is much appreciated. I am a subscriber to KM World Magazine and have also enjoyed its content so we consider this mention an honor. 

Using Chaos Theory Principles to Overcome Information Overload within the Enterprise and on the Web – Part Two - My Enterprise 2.0 Conference Notes

This is the tenth in a series of notes on the Enterprise 2.0 conference in Boston, June 14- 17. This one is different as it covers the session led by myself and by my Darwin colleague, Thierry Hubert, Using Chaos Theory Principles to Overcome Information Overload within the Enterprise and on the Web.  Here is a link to the notes on part one of this session. Here is our description and summary of Thierry’s comments follows.

“With the addition of social media to the already increasing amount of Web content companies are spending more resources trying to make sense of what is happening. In addition, the connections between formal, informal, structured and unstructured information are becoming more difficult to establish. It may only get worse with the advent of auto-generated content and other content “farms.” In this session we will explore and discuss how the application of chaos theory to this issue can help break the silos of information and allow the emergence of meaningful awareness for better decision making.  This new paradigm moves beyond Page Rank to reduce the impact of traditional SEO techniques that elevate low quality content.  We will look at how the visualization of connections between content related to a particular theme can reveal new relationships and help with the discovery and awareness of trends, both anticipated and unanticipated.”

Thierry began but pointing out how we are surrounded by chaos on the Web through the over abundance of content. Page rank was a big improvement over prior search methods. However, it was conceived before the advent of social media. The increased and growing volume of Web 2.0 information is causing search engines to rethink their outdated popularity ranking algorithms by bringing timely social media content to the foreground in an attempt to increase relevance for users. 

In the recent FaceBook privacy controversy, Web users made it clear that they are not so eager to allow search engines to mine their communities’ content for the benefit of search engine companies and other third parties.  As a result, social media content is not rescuing search engine companies’ quest for intimate relevance.   Nonetheless, search is a critical part of the Web experience, but it is no longer the only way to deliver relevant information.

It is since the advent of the Web 2.0 that search engines are no longer capable of elevating recent and relevant information in a timely and contextual manner.  Popularity ranking is corrupted by Search Engine Optimizations (SEO) and requires too much time to elevate pertinent information that has a short shelf-life.  The sad thing is that users are missing meaningful information that is hidden on the 50th-plus page of a search result; they just don’t know what they don’t know since they can’t find it.

To address this challenge Darwin Ecosystem’s Awareness Engine™ applies chaos theory principles to reveal emerging patterns and correlations that accelerate the user’s understanding of what is happening about a given theme of interest regardless of popularity ranking.  We believe this organic and self-organizing model is best adapted to the evolution of today’s Web when compared with the increasing limitation of page ranking offered by search engines.  Although not a replacement for search engines, Darwin’s Awareness Engine highlights a new and missing perspective. Chaos theory does not offer a precise answer, but it reveals a movement or trend as it emerges in real time.

Thierry provided a number of examples of Chaos Theory in operation through the Darwin Awareness Engine. The Darwin interface provides a simple and interactive ScanCloud™(patent pending) that reveals correlated themes for contextual content filtering around your topics of interest, a BuzzTape™ that displays themes that have high content-acceleration within the last 24 hours, and the ability to save attractors (term used in chaos theory) to monitor the evolution of your favorite topics. Below is a sample Scan Cloud.

At the top is the Buzz Tracker ™showing the rising and falling starts. At the top left you can see that it is set to catch what is happening in the past hour. The Scan Cloud ™ shows the top themes emerging in the target sites. The right column shows the stories under these themes with links to the actual story. Now here is a close up of the Scan cloud so you read it better.

In this instance there is a reference to Knoxville in the middle of the Scan Cloud. If you click on it the stories in the right column shift to those related to Knoxville and display two stories about use of social media related to the recent floods in the area.

Looking at what was under Twitter in the Scan Cloud and the two stories below are found among others.

You can link to one of the stories in PC Magazine as displayed below.

Thierry provided a number of examples but this one gives a sample of the basic interface and process. Darwin’s Awareness Engine can provide a new model of information consumption through Chaos Theory that will benefit Web and Enterprise users alike while allowing Web 2.0 authors to be noticed in a timely and relevant context. 

Using Chaos Theory Principles to Overcome Information Overload within the Enterprise and on the Web – Part One: My Enterprise 2.0 Conference Notes

This is the ninth in a series of my notes on the Enterprise 2.0 conference in Boston, June 14- 17. This one is different as it covers the session led by myself and by my Darwin colleague, Thierry Hubert, Using Chaos Theory Principles to Overcome Information Overload within the Enterprise and on the Web.  Here is our description and then summary of my opening comments follows. The next post will cover Thierry’s comments.

“With the addition of social media to the already increasing amount of Web content companies are spending more resources trying to make sense of what is happening. In addition, the connections between formal, informal, structured and unstructured information are becoming more difficult to establish. It may only get worse with the advent of auto-generated content and other content “farms.”

In this session we will explore and discuss how the application of chaos theory to this issue can help break the silos of information and allow the emergence of meaningful awareness for better decision making. 

This new paradigm moves beyond Page Rank to reduce the impact of traditional SEO techniques that elevate low quality content.  We will look at how the visualization of connections between content related to a particular theme can reveal new relationships and help with the discovery and awareness of trends, both anticipated and unanticipated.”

I started by saying that we do not see Chaos Theory as put in practice through our Darwin Awareness Engine™ as a replacement for Google. We see it as a complement. It is an awareness or discovery engine not a search engine.

For example, when I want to find the TV schedule for the NBA playoffs or the Web Site of the Boston Celtics or even the Los Angeles Lakers, or an article I heard about at this conference, I turn to Google.  I use Google when I know what I am looking for.

If I want to find the stories I did not know to look for about a topic of interest, if I want to discover new things, if I want to see the breaking news in real time, the stories generating the most buzz, I turn to Darwin. Chaos theory does not offer a precise answer, but it reveals a movement or trend as it occurs.  I use Darwin when I want to explore any area of interest in more depth.

Now if I go to Google I get a linear list of results rank ordered by its external framework. Google decides what is important for you. Darwin provides an overview of the 100 top themes related to your topic of interest visualized in a tag cloud type structure and allows you to explore the ones that appeal to you. You become the decider and we will show some real time examples.

Rather than taking a linear, deterministic approach to finding content through seeking repeatable patterns on the Web with an external framework like page rank as done by Google, we use Chaos Theory to find non-repeatable patterns in Web content that come and go in real time. Rather than using an external framework, we look for self-organizing patterns within the content itself.

Sometimes external frameworks and measurement have their place. Other times it can be an obsession that gets in the way. For example, Fred Taylor tired to systemize work processes and this approach led to profits in some cases but has alienated most workers. In his Atlantic article, Will Google Make Us Stupid, Nick Carr quoted Google CEO Eric Schmidt, that Google is “a company that’s founded around the science of measurement,” and it is striving to “systematize everything” in the same spirit as Taylor. This can be useful; but it also has its limits. It depends on the use case. Robert Parker has tried to systemize wine reviews and in my view is a clear over step. How can you say with any certainly that one wine is a 94 while another is a 92 and be serious about it?

We offered this video showing Robin Williams in The Dead Poet’s Society discussing the value of a measurement system for poetry. Here is a link to the Measuring Poetry episode on YouTube.

Now I am not saying that Google is like Pritchard on poetry or Robert Parker on wine in that it is useless. Often you have to lay pipe in Robin Williams terms. It is the default home page on my laptop and I use it everyday. However, I do say that its quest to systematize everything and make it serve all content discovery functions is a delusion. 

 Google has its limits. For example if I type in - Sardinian beach bars - into Google I find that my blog post is number one out of 349,000 as it has been for the past five years since I wrote it. I have even used external frameworks myself.

On the same trip my friend and I where sitting at one of these beach bars and over a few glasses of wine devised a measurement system to rate the beaches of Italy. Now if you type rating the beaches of Italy into Google you will find that my blog posts are number one and two out of 691,000.   

Of course I offer these examples with great humility. I have no delusion that I offer the best content here or that I am the world’s expert. I was just better at speaking Google’s language than the others on the topic.   This is a problem when you use an external framework. People learn how to game it for good and bad reasons. SEO and spam are byproducts of Google.

I must confess that I have done some consulting over the past three years helping firms get their blogs on the front page of Google, using respectable methods I add.

If you use chaos theory to let the content self organize there is no external framework to game or approach through legitimate methods. There is no SEO or even spam as it would serve no value. 

To set the stage we went back to the creation of the text we use to create content. One of the greatest information technology inventions remains the phonetic alphabet. The Greeks came up with it in the early 8th century BC. The same twenty-four letters have been used to write the Greek language ever since. It is the first and oldest alphabet to note each vowel and consonant with a separate symbol.  This invention provided great flexibility and unprecedented accuracy of linking symbol to word and thus on to thought.  Now we moved from relying on informal communication through conversations to formal documents.

One of the great flexibilities of the phonetic alphabet or text was the wide range of vehicles that can be used to convey text. Most of the technological advances since the work of the early Greeks have been on the vehicles to convey text, such as paper, and the means to produce these vehicles, such as the printing press, rather than the alphabet, itself.  Text remains king on the Web and the darling of search engines.

In the past, the effort to create text acted as a filter but a series of inventions such as the printing press cut into this difficulty reducing the barriers to creating formal context, generally for the good of most

Now with Web 2.0 the barriers to creating permanent content through text have been mostly shattered. You still need a computer or a mobile phone or some other electric device and access to the Web.  But much of what is being added is closer to the informal conversations of the pre-text world, than the type of content normally taken to print. This has both good and bad consequences. As Plato said in reaction to a possible over-reliance on documents, conversations can clarify intensions. On the plus side we have much more to look at. One the negative side we have much more to look at.

Next I showed the evolution of content delivery channels.  In 2008 more content was created through the Web than in the history of content. The same thing happened in 2009.  We are now in danger of being overwhelmed by content and sinking into chaos. Why not turn to Chaos Theory to help pull us out? Instead of imposing an external framework to attempt to make sense of content, why not let content self-organize itself. Instead of looking for repeatable patterns, why not look for correlations between content to discover new relationships beyond what you where initially looking for?

My colleague Thierry Hubert now took over to explain how this can work.  His comments will be in tomorrow’s post. 

Rising Above the Over Quantification of Content

I recently posted a four part series on the Darwin Discovery Engine Blog on Rising Above the Over Quantification of Content. I will not repeat the whole series here. Below are links to the first three posts. Then I offer the concluding post, Rising Above the Quantification of Content: Part Four Google vs. Darwin. I encourage you to look at the first three before moving to this final episode. 

Rising Above the Quantification of Content: Part One: Parker vs. Piaget

Rising Above the Quantification of Content: Part Two: Pritchard vs. Williams (aka Keating)

Rising Above the Quantification of Content: Part Three: Powerpoint versus Text and Conversation

In this last part of the series on the value of providing people the opportunity to exercise their own cognitive powers to make sense of content, see the connections between content, and decide what is valuable to them. I have reviewed some efforts to over measure topics beginning with Robert Parker on wine and J. Evans Pritchard on poetry and looked some effects of media on cognition by exploring what PowerPoint may be doing to us as reported in the New York Times.  Nick Carr addresses similar questions in his Atlantic article, Is Google Making Us Stupid? He provides some anecdotal evidence but admits that the proper scientific research has not been done.

I do not feel that Google actually makes us stupid but also do not feel that it is not going to be the key to understanding the universe that seems to be the quest for Google.  I am also not suggesting that Darwin is here to replace Google. It simply complements it and does things that Google is not so good at. Google is a search engine and it helps you find things you are looking for. I keep using it to write this post. Darwin is a discovery engine and it helps you explore a topic and develop a greater awareness of its context and uncover things about the topic that you did not know to search for. I have written about the Google question before (see Will Google Make Us Stupid? in the FastForward blog). I concluded that like any information tool, the final responsibility for determining the value of the content lies with the person using it. Darwin is designed to give the user a greater opportunity to make these decisions.

I do agree with Nick Carr that media and knowledge related technology effect cognition, that the means we use to work with information have effects on our thought process. I underplayed this aspect of the Google question in my FastForward post. Our communication channels, whether they are words, charts, PowerPoint, or Twitter, provide structure to our thoughts and help shape them and our subsequent thinking. This was the focus of my psychological research in the 70s, first at the University of Toronto and then at Harvard.

One clear example of the effects of media on cognition comes from comparing spoken language with sign language. For example while errors in speech often reflect the structure of spoken language, the neuroscientist Ursula Bellugi found that errors in sign language reflect its parameters: space, direction, and movement. Likewise the poetry of spoken language reflects the rhythms of sound while the poetry of sign language is based on the rhythms of visual movement. A communication medium does more than convey information. It helps to structure your thoughts.  In the last post in this series I discussed a bit about the effects of the phonetic alphabet on thought.

Carr offers a clear example of the effects of knowledge related technology when he discusses how the mechanical clock, which came into common use in the 14th century, changed aspects of thought. He refers to the work of Lewis Mumford, the historian and cultural critic who described how the clock “disassociated time from human events and helped create the belief in an independent world of mathematically measurable sequences.” The “abstract framework of divided time” became “the point of reference for both action and thought.”

So both technology and communication media can effect thought. The Web is a mixture of both. Carr wonders about the effects of using the Internet on our cognitive abilities. He writes, “Never has a communications system played so many roles in our lives—or exerted such broad influence over our thoughts—as the Internet does today. Yet, for all that’s been written about the Net, there’s been little consideration of how, exactly, it’s reprogramming us.”

Carr considers Google to be the “Internet’s high church” and he goes into great depth on how Google’s obsession is with measurement, like Fred Taylor, Robert Parker, and J. Evans Pritchard. He quotes the Google CEO Eric Schmidt, that Google is “a company that’s founded around the science of measurement,” and it is striving to “systematize everything.”  Sound familiar?

Carr goes on to write, “It seeks to develop “the perfect search engine,” which it defines as something that “understands exactly what you mean and gives you back exactly what you want.” In Google’s view, information is a kind of commodity, a utilitarian resource that can be mined and processed with industrial efficiency. The more pieces of information we can “access” and the faster we can extract their gist, the more productive we become as thinkers.” 

He also notes that the economic model of massive page views drives them to encourage us to skim from page to page, not stopping to explore and contemplate.  Google provides a linear list of results organizer according to Google’s concept of what is useful to us. Google tries to be the decider on value. Carr is concerned that this is causing us to move away from reflective thought and focused reading for a deeper meaning.

Darwin takes an opposite view. It views the Web as too massive to understand and human reasoning too complex and individualized to fully progam. Rather that drawing inspiration for Fred Taylor’s time and motion studies and attempting to understand and control the universe of content on the Web, Darwin is inspired by Edward Norton Lorenz’s work on Chaos Theory in mathematics and weather prediction. Darwin does not try to impose an order on content, it lets content self-organize. During the 1950s, Lorenz became skeptical of the appropriateness of the linear statistical models in meteorology, as most atmospheric phenomena involved in weather forecasting are non-linear. Instead he proposed the concept of attractors that operate within a dynamic system that evolves over time in a complex, non-repeating pattern.  This is the complete opposite of Fred Taylor’s repeatable time and motion approach.

We think the two approaches complement each other. The Darwin approach does not require human intervention to be organized; it is self-organizing based on the content itself and therefore more representative of what is happening as it occurs, and respectful of each unique context. It also eliminates the possibilities of search optimization and spam.  Instead of providing a linear list, Darwin offers a Scan Cloud of the top 100 themes related to your topic of interest (see below).  Your chosen topic acts as an attractor to collect related content. Darwin allows you to move within the Scan Cloud to explore the related themes that are of interest to you and the Scan Cloud dynamically changes to reflect your movement. 

A list of the actual content within each theme is found next to the Scan Cloud (see below) and it dynamically changes to reflect your movements.  Darwin does not try to understand the universe of content nor does it try to understand your intentions or values. Instead it helps you to achieve your own understanding of what is happening in the world of Web content.

Darwin promotes exploration, contemplation, and discovery. We do not make any claims on its lasting impact on thought but it does address some of the concerns that Nick Carr raises. We invite you to contact us to take a look at our Darwin web site. 

 

Using Darwin to Track Social Media Buzz

We have set up a special instance of the Darwin Awareness Engine ™ to track social media buzz. We targeted around 300 of the top social media bloggers and the relevant news sties. I am now monitoring it and have been finding interesting things to blog and tweet about. Below is the overall screen. At the top is the Buzz Tracker ™showing the rising and falling starts. At the top left you can see that it is set to catch what is happening in the past hour. The Scan Cloud ™ shows the top themes emerging in the target sites. The right columns show the stories under these themes with links to the actual story.

Now here is a close up of the Scan Cloud so you read it better.

I noticed the reference to Knoxville in the middle of the Scan Cloud. I clicked on it and this caused the stories in the right column to shift to thoise related to Knoxville and I found the two stories about use of social media related to the recent floods in the area.

Then I looked at the what was under Twitter in the scan cloud and found these two stories, among others.

I linked to one of the stories in OPC Magazine as you can see below.

I have found this to be a useful additional way to monitor social media buzz and explore topics that interest me, especially ones that I was not looking for. 

Reflecting on the Edges of Time

A few weeks ago I read an interesting blog post, The Edges, by Edward Harran, that I found through Twitter from Ross Dawson.  He began with the idea that “The past, present, and the future are all one and the same” and added, “new things always happen on the edges.” However, he goes to add that it is hard to see the new things because you do not know they exist and do not know where the edges are located.

Edward summarizes this issue as follows: 'The Future' is the continual emergence of the edges in the present. 'The Past', similiarly, is the continual understanding of the edges that were discovered and, that which we are now, attempting to put into context.
'The Present' is, thereby, in the intersection of these two ongoing investigations. They are all built and interwoven into each other.

Edward concludes with a quote of Eckart Tolle, "The present is the only thing we will ever experience. The now." He adds, “so we better start understanding it.”

This is our mission at Darwin. We try to provide the context for the present in real time thorough our Buzz Tracker™ and Scan Cloud™ so you can better understand the past, anticipate the future, and better live in the present.

Years ago I wrote a poem that contained these lines within it,

A baby cries and laughs

through the eternity of every second,

existing in a world devoid of tense,

until he learns to live for tomorrow

by rules from the past.

 

And belief is built on belief.

 

When is the present?

It becomes the past before you can grasp it…

 

…and Bert and Walter sat in the corner,

sharing their wine and worldly views.,

and talking in the past and future tense… 

While real time is actually a relative statement, understanding its context will help you better find the edges.  Thanks to Eddie for bring me back to this topic. 

Creating a Sustainable Ecosystem for Community News Media

Today the traditional print news media has been impacted by the rise of the Web and social media, in particular. However, this emergence also represents an opportunity for more robust news gathering, increased community out reach, and better measurement of impact from both print and Web channels. 

At Darwin we have been talking with several news organizations on ways to expand their community connection.  These conversations inspired a recent three part series on creating a sustainable ecosystem for community news media. I will not repeat the content here but encourage see the complete series on our Darwin Awareness Engine blog.  Here are the three posts.

Creating a Sustainable Ecosystem for Community News Media - Part One: Providing Relevance and Creating Loyalty

Creating a Sustainable Ecosystem for Community News Media - Part Two: Finding Relevant Local Content from the Web

Creating a Sustainable Ecosystem for Community News Media - Part Three: Making Awareness and Discovery the pulse of Community News with Darwin 

In summary community news organizations can benefit from measuring and monitoring of the evolution of community themes over time or dimension by:

1. Tracking and measuring the impact of the Community News Media’s programs’ actions in the community and/or on the broader Web

2. Creating alerts based on significant variations within the designated themes of interest spikes

3. Advancing analytics for pattern recognition and prediction

With these benefits, the community news organization can achieve better awareness of emerging themes to improve its interaction within the community and provide highly targeted advertisement to support its financial targets.

Are the Next Takeoff Technologies Evident Now?

Here is another of the Pew Research questions "future of the Internet" survey. I wrote about Andy Oram’s response to the question: Will Google Makes Us Stupid?  Andy responded to more questions including: Are the next takeoff technologies evident now?  Andy said yes. I would not be so bold to say that I can predict that all of them are evident but many are. I liked his list. Andy also qualified his thoughts by saying this was a starter list.

Andy's list includes, “more powerful mobile devices, ever-cheaper netbooks, virtualization and cloud computing, reputation systems for social networking and group collaboration, sensors and other small systems reporting limited amounts of information, do-it-yourself embedded systems, robots, sophisticated algorithms for slurping up data and performing statistical analysis, visualization tools to report the results of that analysis, affective technologies, personalized and location-aware services, excellent facial and voice recognition, electronic paper, anomaly-based security monitoring, self-healing systems--that's a reasonable list to get started with.”

Apple is certainly banking on the first two. At Darwin we are particularly pleased with another two of these: sophisticated algorithms for slurping up data and performing statistical analysis, visualization tools to report the results of that analysis. This is what we do. The Darwin Awareness Engine ™ helps to uncover relevant content to your interests and the themes associated with these topics.  Features such as our Buzz Tracker ™ and our Scan Cloud ™ help to visualize this emerging content and the relationships between correlated content. 

Join our Enterprise 2.0 Conference 2010 Session in June in Boston

 I was pleased to recently learn the session proposal, Using Chaos Theory Principals to Overcome Information Overload within the Enterprise and on the Web, that I submitted with the my Darwin colleague, Thierry Hubert, has been accepted for the Enterprise 2.0 conference in Boston on June 14-17. I have presented at this conference for the past three years and I am very pleased to do it again. I think this is one of the best conferences on enterprise 2.0 and I always get to see so many virtual colleagues in person  Here is a post on the panel session I participated in last year, Thoughts on Twitter and Business for Enterprise 2.0 Conference. Below is the description for our session. We hope to see you there.

With the addition of social media to the already increasing amount of Web content, companies are spending more resources trying to make sense of what is happening. In addition, the connections between formal, informal, structured and unstructured information are becoming more difficult to establish. It may only get worse with the advent of auto-generated content and other content "farms." In this session we will explore and discuss how the application of chaos theory to this issue can help break the silos of information and allow the emergence of meaningful awareness for better decision -making.  This new paradigm moves beyond Page Rank to reduce the impact of traditional SEO techniques that can elevate low quality content.  We will look at how the visualization of connections between content related to a particular theme can reveal new relationships and help with the discovery and awareness of trends, both anticipated and unanticipated. 

Thierry and I are also speaking the Enterprise Search Summit in New York May 11 on Deriving Order From Chaos Through Discovery and Awareness.

Darwin Discovers Emerging Buzz Over UCLA Tuition Hike Protests

Here is an example of how the Darwin Awareness Engine™ can uncover breaking news before it hits the mainstream press.  We have the concept of rising and falling stars as topics gather or lose momentum on the Web.  These rising and falling starts are displayed in a ticker tape type across the top of the Darwin screen.

Recently, Thierry Hubert notices an acceleration or rising star on the term – tuition hike.  He looked at the formal mainstream media and found nothing. Then he looked at informal media and found a lot of content and variety of related themes on the Darwin Scan Cloud™.  For example, there was a video of UCLA students protesting the tuition hike at their school.  There are several other sites with similar videos. It appeared to be a coordinated by the UCLA students through social media. Here is link to a video of the Darwin screens and results hosted on Vimeo.

We found a similar example recently, when we saw that the Latino community was organizing through social media to get Lou Dobbs. This social media effort appeared through Darwin before it was picked up by the mainstream press. 

Will Apple iTunes Impact the Business Model in Television, Like Music?

Things continue to move fast in the TV sector. Apple and other Web apps have certainly impacted the music industry and print publishing.  Now there is a move to impact television by changing the business model. Instead of subscribing to cable for a fixed fee to get a lot of stuff bundled, Apple wants to sell TV programs one at a time, like they do with music, for a low cost per program. A recent Fats Company article best that they will succeed (see: Apple's iPad Disrupting the Network TV Business Model? I'd Buy That for a Dollar!).

Naturally, the TV industry is exhibiting broad concern over this plan. While they're want to get on the Apple's iPad bandwagon as a TV player, they are worried that lower prices will cut into their revenue streams. Now TV has its markets to protect but there are also 125 million registered iTunes users, representing a huge potential market, waiting to be tapped by someone. The main worry on the TV side is that production is expensive and each show represents a huge investment in time and money. Can they get their money a buck at a time?

The FastCompany article, which builds a story in the New York Times, suggests that they would be fools to not do this. If they do not offer programs at attractive prices, people will just steal the shows as it is not that complex to do in the digital world. If the price is right, it will convince most people not to do to the trouble and potential risk to resort to piracy.  It will be very interesting to see how this shakes out. We feel that Darwin can help the winners, whoever they are, with both news gathering and impact analysis regardless of the winner as TV moves closer to the Web (see Monitoring and Measuring the Impact of Public Service Media Part Two: Addressing the Issues). 

BBC Elevates Social Media for News Creation and Monitoring

According to the Guardian (see BBC tells news staff to embrace social media).  BBC news journalists have been advised to use social media as a primary source of information by Peter Horrocks, the new director of BBC Global News. He took over last week and said it was important for editorial staff to make better use of social media and become more collaborative in producing stories. They quote him, "This isn't just a kind of fad from someone who's an enthusiast of technology. I'm afraid you're not doing your job if you can't do those things. It's not discretionary."

I would agree. Getting material for articles is one of several ways that traditional television news media needs to make use of social media to survive the social media onslaught. For BBC news editors, Twitter and RSS readers have now become essential tools and aggregating and curating content with attribution are essential skills. In addition, BBC's journalists have to integrate and listen to feedback for a better understanding of how the audience is relating to the BBC brand.

At Darwin, we have been talking with a number of major traditional media organizations about using the Darwin Awareness Engine™ to help with their news harvesting efforts. It allows you to see what is going on around a topic and find the unexpected, as well as the news breaking in real time.

The BBC also created a social media editor post in October. This is another related trend I have reported on here (see Mainstream News Take on Social Media Directors.  The Guardian concludes by noting that as technology is changing the nature of journalism, the BBC is trying to keeping up with the pace. Horrocks is quoted again, "If you don't like it, if you think that level of change or that different way of working isn't right for me, then go and do something else, because it's going to happen. You're not going to be able to stop it."

 

 

Ross Dawson's Top Ten Trends for the Upcoming Decade

Here in more on what to expect in 2010 but it extends to the decade.  Ross Dawson posted his top 10 trends for the 2010s and called it the most exciting decade in human history. I wonder if we can say that at the start of every decade with increasing technology. I still look fondly back at the 1960s but more for personal reasons. I am spoiled and would not want to go back to 60s technology. 

In this post, I will focus on the first two in this post as they relate to the Darwin Awareness Engine’s mission. The first trend is Information Intensity. Ross wrote that, “we will soon consume more media than there are waking hours, by virtue of multi-channeling at most times. Billions of people and places will be media producers, including video streaming from most points of view on the world. We are just at the dawn of an incomprehensible daily onslaught of news and information – some valuable, much useless.’

The topic of information overload has been around for some time, probably since the first enhances to communication technology thousands of years ago. But now with all the user generated content it takes on new meaning. More content was created through social media in 2008 than in all of prior history.  The Awareness Engine helps you find what is relevant to your topics of interest and then lets you decide where to focus within these topics. You do not have to know what you are looking for as in traditional search.

The second trend, Intelligence, is a partial remedy to the first. Ross wrote that while we are swamped by information, “we are now creating a collective intelligence that will filter and respond to what is worthwhile. Reputation measures will drive who we meet, do business with, and date. Machine translation will enable the people of the planet to communicate…”

Darwin offers another way to harness the collective intelligence. Instead of using reputation measures, it looks at relevance and correlation.  Reputation, pioneered by Google, is useful but it can be spammed and it tends to promote the already popular, rather than the unexpected, but useful.  Instead of waiting for linking to occur, the Awareness Engine looks at what is emerging in real time that is relevant to your topics of interest. It does not replace reputation measures but offers an alternative.

Peter Cashmore offered a similar prediction about how your friends will serve as filters for content. Instead of indirectly through reputation, he noted the more direct methods available through channels such as Twitter and Facebook. Darwin allows you to also become your own curator or producer of filters. You can set attractors on your topics of interest: people, places, concepts, and more. Then you can see what emerges. This can be especially valuable for niche areas. You can create your own online magazine.

Ross also offered 10 TENsions That Will Define 2010. One was on a topic we have frequently covered here, Death of Media – Birth of Media. Ross wrote,
”Literally hundreds of newspapers around the world have shut their doors in 2009. Broadcast TV is struggling. Advertising has slumped. Yet as traditional media staggers, a new world of mobile media, social media, video everywhere, and new business models are opening a new era in which media is at the center of the economy.” We would add that traditional media can survive if it learns to make creative use of the new media and not simply try to port old methods to the new media.  Examples include, Radio’s Air America, New York Times, and Los Angeles Times.

Comparing Google Web Search and the Darwin Awareness Engine™

I want to share this post I first wrote for the Darwin Awareness Engine blog last week to see what you think and what questions you might have. Google has been the dominant web search engine for years now for good reason. It is the default opening screen on my laptop and I use it multiple times every day. It brought the innovative concept of Page Rank to gauge popularity and provide us with useful, quality content.  The Darwin Awareness Engine™ brings a different approach through its use of Chaos Theory and the concept of correlated content. I am certainly not attempting to equate these innovations but I want to attempt to better explain what the Darwin Awareness Engine does by comparing it to the well-known Google Web Search. First, let’s review what Google does.

You put a search term or phrase into Google. It brings back a list order by its algorithm based on reputation (link popularity), relevance (based on content), and currency.  Google decides what is important for you to see. Lately Google has started to try to better understand what you do to bring more focused and personalized results. However, they are still the “decider” and the list is one dimensional with the recent addition of some social media sidebars. 

This works very well but it can be subject to spammers who attempt to manipulate the dimensions of reputation and relevance through nefarious means.  There has been some concern of the impact on legit content (see ReadWriteWeb on Content Farms: Why Media, Blogs & Google Should Be Worried and Tech Crunch on The End Of Hand Crafted Content). There are also the more honest search engine optimization people (SEO) who are simply trying to get the best returns for their honest content.  For example, I help companies with their blogs and one of the services I offer is SEO for legit content.

 Applying SEO to Google can lead to some interesting results. A Google search on - beach bars Sardinia – finds a blog post on beach bars in Sardinia I did based on a vacation trip as number one out of over 250k results. As far as I know it has held that position for the past four years. Now granted I did original field research with a friend and it is part of a series, Rating the Beaches of Italy, which also remains number one in Google (out of 800k) for that term. While my content is legit, I have to admit that I am far from the best expert on Italian beaches and their bars.  The listing on Google is more the result of using good SEO to get there and the fact that it is a niche topic.

Google is trying to determine the proper order of the universe of content. It does a great job but it is up against an impossible task according to Chaos Theory. It will not achieve perfection and it will have to try to stay ahead of spammers and coexist with legit SEO along the way. Darwin’s Awareness Engine does not attempt this task. It accepts there is no order in the universe of content and let’s you be the decider. It just tries to expose you to the different dimensions and themes in your topics of interest. It tries to make you aware of what is going on so you see trends and find things you were not aware of.

The Darwin Awareness Engine uses the concept of attractors (based on Edward Lornez’s attractor concept – see math model in illustration w via Wikipedai) to let the content order itself through correlations of related   themes that emerge. Instead of a list ordered by Google you get a Scan Cloud™ that displays the content related to your chosen topic that is organized by the themes that have emerged that correlate with your content.  Under each theme is a set of related content and you can drill down to this content. You can decide which themes to explore. You can also decide whether you want to see the results form informal (blogs, etc.) or formal content sources (mainstream news sites, etc.).

If you are interested in climate change you see content organized under such themes as - threats - or - hoax - depending on your view and, of course, see what the other side is saying. You can also see what the English speaking press is saying about the Copenhagen conference or what the French speaking press is sawing by looking under the English or French spelling of the term Copenhagen. 

You can see content emerging in real time that is not order by popularity links and does not have to wait for the links to be created for it to emerge in search results.  This allows for unanticipated results that might be buried in the back pages of a Google search for days, if not forever. For example, when looking at content related to the prime minister of Quebec we found coverage of the current initiatives in electrical service to the Canadian Maritime provinces. This was expected. We also found some unexpected emerging concern by the oil companies over this move as the theme – oil – appeared in the Scan Cloud. This was not expected.

The Darwin Awareness Engine is not designed to replace Google or other search engines. These tools are largely designed to help you find what you know you are looking for. With many search engines you can narrow this focus with filters but these are filters that you select. In contrast, the Darwin Awareness Engine is designed to provide an alternative way of letting you discoverer what is happening in your areas of interested, even the unexpected. Here is more. 

Text Analytics Offers a Discovery Channel to Make Better Sense of Content

Socrates said that, “the unexamined life is not worth living.”  In perhaps a parody of this statement the Forrester report, Text Analytics Takes Business Insight To New Depths, states that “unexamined content is a wasted opportunity.” They look at an array of tools that go beyond simple search. This might be a good thing as a Google search on the term, the unexamined life is not worth living, turns up a number of term paper services in its top ten.  The Forrester team of Leslie Owens with Matthew Brown, Sara Burnes, and Peter Schmidt conclude through the report subtitle that “An Obscure Technology Has Found Its Killer App.” I would certainly agree and appreciate receiving a review copy of this report. 

The authors note that customers, employees, and competitors comment on products, personalities, and companies in increasingly public places, like Twitter and discussion forums. Forrester that online ratings and reviews topped the list of trusted material. The abundance of news and commentary on the Internet could contain actionable knowledge for your business. This is not simply a Web issue as internal sources like emails and call center notes are full of product suggestions, feedback on competitors, and thoughts on the market.  With the rise of enterprise 2.0 and related social media data, the opportunities for data mining and information overload are expanding. To make sense of all this content, businesses are turning to text analytics tools.

The team concludes that this “little-known technology has a compelling value proposition: extract meaning out of large quantities of text by mining, interpreting, and structuring information to reveal hidden patterns and relationships.” I am not so sure about the little known part but I agree with the rest of the statement.  Perhaps I am biased because I have been talking with a lot of vendors in this space.  One of the great opportunities of both Web 2.0 and enterprise 2.0 is all the conversational content that it creates. You would be losing a good bit of the value it you do not look at this content but you need good tools to do it.

The team found that here is too much content to review by hand, It is hard to separate a signal from noise, and here is no way to know what questions to ask.  They add that when you don’t know what’s significant about a set of content, you can’t investigate it with a search engine. The report reviews a number of the major text analytic players and should be useful for those making decisions in this space. Text analytics software typically includes four key components: entity extraction, categorization, relationship mapping, and sentiment analysis.

The authors caution that “until recently, text analytics software was obscure and academic, used primarily by early adopters in the life sciences field… Text analytics is a framework and a process. It is not always necessary or possible to deploy text analytics as a single, linear solution from a single vendor.” It does seem that many of the solutions are complex and are designed to be used by professionals, often with subject matter expertise in linguistics and statistics. There is much more and you can obtain the report at the Forrerster web site.

As a post script to my comments on this useful report I want to say a bit about Darwin Ecosystem. Readers of this blog will know that I am connected with Darwin. The Darwin Awareness Engine™ offers a different approach to the same issue. Using chaos theory it uncovers the emerging themes about a topic, a form of categorization. However, it operates through a simple interface that is designed to be accessible by a wide range of people. The Scan Cloud™ visualization allows you to immediately see the relationships between content groupings in real time within your chosen theme and draw your own conclusions as to what trends are occurring.  It is another way to get into content discovery. Putting The Darwin Awareness Engine in the hands of experts, will better equip these individuals to discover new trends in their field. However, ii is also accessible to the average business user so they can have a greater awareness of their areas of interest. 

Web Trends to Watch in 2010 from Peter Cashmore

Here is another set of 2010 predictions. Peter Cashmore, CEO at Mashable provided 10 Web trends to watch in 2010 to CNN. They all seem likely but I want to focus on four that relate to Darwin Ecosystems Awareness Engine™: real-time ramps up, content 'curation,' cloud computing, and fame abundance, privacy scarcity.

Peter commented that, “sparked by Twitter, Facebook and FriendFeed, the real-time trend has been to the latter part of 2009 what "Web 2.0" was to 2007. The term represents the growing demand for immediacy in our interactions. Immediacy is compelling, engaging, highly addictive ... it's a sense of living in the now.”  Darwin’s Awareness Engine™ allows you to set the time frame you sample for relevant content. It operates in the immediacy of real time and can go back to the last few hours, days or weeks.  You can see what is emerging and where it is emerging. For example, the talk about Lou Dobbs’ actions a few weeks ago first emerged on the informal media of blogs before it appeared on the mainstream media.  By looking separately at informal and formal media we could see the hot topics on each.  Often the topics on informal media are predictors of what soon will hit the mainstream.

Peter wrote that, “The Web's biggest challenge of recent years is that content creation is outpacing our ability to consume it: "Information overload" has become an increasingly common complaint… In 2008, the answer revealed itself: Your friends are your filter… Increasingly, your friends are becoming the curators of your consumption.”  I certainly agree here and Twitter has served this role for me.  Much of what I write about on my blogs comes from my Twitter friends, including the link to Peter Cashmore’s predictions. Darwin allows you to also become your own curator. You can set attractors on your topics of interest: people, places, concepts, and more. Then you can see what emerges. This can be especially valuable for niche areas. You can create your own online magazine. It can cover a much greater array of source than Google News and it will show the relationships between news items that emerge rather than simply displaying them. You can adjust your filters to further focus you’re your curation efforts. We see content curation as one of the major applications for the Darwin Awareness Engine™.

Peter added that, “Cloud computing was very much a buzzword of 2009, but there's no doubt this transition will continue.” We are a cloud application.

Finally, Peter wrote that, “Warhol was right: Fame is now abundant. Social media has birthed a galaxy of stars in thousands of niches: We're all reality stars now, on Facebook, Twitter and all the myriad online outlets where we hone our personal brands.” With Darwin we have the feature of rising stars and fading stars. You can see what topics are merging on the general Web or within you niche. You can also see the topics that are fading in importance.  These results are not based on popularity and are hard to push in spammy ways. The importance is based on the substance of the content itself and how it relates to other content within the target area, the whole Web or your subset.

I am pleased with Peter’s predictions for 2010 and look forward to the potential of Darwin playing a role within them. 

Mainstream News Take on Social Media Directors

Dirk Smillie wrote at Forbes.com in Journalism's Hottest Job that there are now about 200 social media directors at newspapers, book publishers, magazines and television news stations. Most started in the last two years. He writes that, “Rarely offline, their job is to tweet, ping, blog and friend-find throughout the day, building and interacting with new audiences, promoting media brands and sometimes breaking news.”  I have been talking about the relationship of social media and mainstream news for a bit (see Social Media Does Not Replace Traditional Mainstream Media and Social Media Helping TV Stations Attract More Viewers and It Could Change Way TV News Created).

Dirk goes on to question whether simply tweeting the news is a money maker. He points out that most newspaper RSS feeds don't carry ads; neither do blog posts or tweets that break news and adds that Lynn D. Johnson, senior vice president for social media at the Advertising Research Foundation, says news organizations aren't earning "anything significant" from social media. This is where the mainstream news needs to get creative. They should not simply reproduce the old media content in the new media. For example, social media can be a useful means to drive traffic to your regular, ad laden channel. A Seattle TV station took over the top TV news ratings spot when its news staff tweeted to establish a better connection to their audience.

Dirk concludes his post with a quote from Woody Lewis: "Whether news organizations believe in its potential to make money or not, they see the train leaving the station and don't want to get left behind. They know that social media is where users want to be." Woody writes the blog, "Save the Papers," which debates how mainstream media can use social media to survive. They certainly need to be more creative in how they broadcast their content with all the new options open. They need to lead readers and viewers to revenue producing channels as well as add revenue to the new channels.

There is another way that mainstream media can use social media. They can better mine it for breaking news and trends. Using the Darwin Awareness Engine ™ we observed a few weeks ago that social media was all over the Lou Dobbs story at least a week before the mainstream media.  Social media creates a vast amount of interesting content. For example, as early as 2005 news people were turning to bloggers for story ideas.  The Awareness Engine can provide greater control and flexibility over how you mine this data. 

Reminder on Our KM Forum Session This Week

Thierry Hubert and I will speak at the December 17 meeting of Boston’s KM Forum.  We will discuss:Deliver Us from Web 2.0 Content Chaos: Activity, Attraction and Personal Knowledge Create the Awareness Engine™.  Here is the session description.

The presentation will share with the audience a practical application of chaos theory for creating knowledge focal points from the chaotic mess of content resources distributed across the Web. Using a couple of examples, one from the Internet where socially generated content is exploding, and one from the enterprise where social tools are also contributing their own content bloat, We will discuss the ideas that are driving a new product idea at Darwin Ecosystem. The examples will illustrate how activity can make information more discoverable, and easier to explore through various visualization techniques. These techniques open the door for more self-discovery through intuitive recognition of patterns. They will help the explorer to find relationships among content objects and people that would not have been found using traditional search methods.

We hope to see you there. It will be from 4 to 6 PM at the Microsoft offices, 201 Jones Road, Waltham, MA. Here is the announcement post on the KM Forum blog with directions. Here is more about Darwin Ecosystem and how it works. 

Our Upcoming Session at Boston KM Forum on Chaos Theory and Information Discovery

Thierry Hubert and I will speak at the December 17 meeting of Boston’s KM Forum.  We will discuss: Deliver Us from Web 2.0 Content Chaos: Activity, Attraction and Personal Knowledge Create the Awareness Engine™.  Here is the session description.

The presentation will share with the audience a practical application of chaos theory for creating knowledge focal points from the chaotic mess of content resources distributed across the Web. Using a couple of examples, one from the Internet where socially generated content is exploding, and one from the enterprise where social tools are also contributing their own content bloat, We will discuss the ideas that are driving a new product idea at Darwin Ecosystem. The examples will illustrate how activity can make information more discoverable, and easier to explore through various visualization techniques. These techniques open the door for more self-discovery through intuitive recognition of patterns. They will help the explorer to find relationships among content objects and people that would not have been found using traditional search methods.

We hope to see you there. It will be from 4 to 6 PM at the Microsoft offices, 201 Jones Road, Waltham, MA. Here is the announcement post on the KM Forum blog with directions. Here is more about Darwin Ecosystem and how it works. 

Social Media Does Not Replace Traditional Mainstream Media

There was a great cartoon in the Boston Globe in early 2005 on the “attack” of bloggers on mainstream media. It showed the barbarian bloggers storming the castle labeled mainstream media. I often use it is blog sessions. However, while it was the view of many then, it conveys the wrong message but some still have this view.  

Recently I attended Webcom 2009 in Montreal and talked about blogging in the age of twitter.  More on that later.  This event attracts a large number of traditional media people and there was a lot of discuss about old versus new (or social media). I attended an excellent session I wrote about on this blog on Who killed the Rocky Mountain News? from John Temple who was the last managing editor of one of the first big papers to fold.  While John admits that many mistakes were made that led to the paper demise, he has learned a lot from the experience and offered some excellent suggestions.

Among other things John said that, “if you want to compete in a medium, you have to understand it.” They built a great web site with no SEO considerations and no one could find it. Another lesson: measure, measure, measure. The Web allows you to measure, take advantage of this. Perhaps most important of all he said that traditional media needs to do R&D to find innovative new offerings that take advantage of the new Web and not just try to get more money for current offerings.  The “us” versus “them” mentality needs to be trashed.

I was reminded of a session that I attended at Harvard’s JFK School of Government in early 2005. On one end of the long table were managing editors of several large papers, including the New York Times. At the other end were some of the big time bloggers of the time such as Dave Winer, Dave Weinberger, and Jay Rosen.  It was like the Globe cartoon I opened this post with. The newspapers said that they spend millions putting reporters out in the field and fact check extensively while the bloggers attract big followings hanging out in their basement in their pajamas writing on their laptop. Dave Weinberger objected to the comparison. He said that the bloggers were just having conversations about what the news reported, among other things.  It was not the same thing. I would agree.

Blogging has opened up a new channel of communication. If the older ones wake up to this they can see how to effectively take advantage of the opportunities. The New York Times has done as good a job as anyone in this area since that 2005 event.

The recent Technorati 2009 survey of the blogosphere also touched on this topic.  It said that, “despite being perceived by some as enemies of the traditional media, bloggers actually carry a journalistic pedigree. 35 percent of all respondents (bloggers) have worked within the traditional media… and the true overlap reveals itself in the 27 percent of respondents who both blog and work in traditional media.”

I actually got started blogging by pitching an article on the topic to a print trade publication early in 2004. I have to confess that I had only heard about blogs the week before. When I sold the piece I started a blog as part of my research and started attending Dave Winer’s blog evenings at Harvard law School. I got hooked and write for many blogs now and still occasionally for traditional media, as well.

Thierry Hubert, who also attended Webcom 2009 wrote, about the discussions I alluded to at the start this post in Webcom 09 - The Misplaced Fear of the Mainstream Media.  He wrote that “The battle is not one of formal versus informal, but one where formal content providers need to listen and engage in the dialogue.”  He adds that at the same time traditional media need to hold on to their charter as “leaders and champions of valued and verified information.” They need to participate in the new social media without losing site of what made them valuable in the first place. 

There can, and should, be a symbiotic relationship between traditional media and the new social media.  Traditional media has given me a lot to blog about over the past five years. I look forward to more and hope that it takes the steps to survive by better engaging in the new Web. 

Here is a recent example of innovative collaboration between social media and traditional media. The Seattle Times recently invited editors from some neighborhood blogs to question mayoral candidates for a town hall-style debate organized by the newspaper and its news partner, KING-TV. The inclusion of the editors underscores a new partnership forged between the city's well-established newspaper and neighborhood news blogs. 

 

We Have Started Twitter Lists on @darwineco

As I have written before, I am involved with Darwin Ecosystem and its Discovery Engine Blog as well as its @darwineco Twitter efforts.  We recently started a number of Twitter Lists.

A lot has been written about Twitter Lists as they open up a number of possibilities. Here are some excellent uses by the news media, Creative Blend of Mainstream Media and Social Media.  The National Hockey League is using Twitter Lists to build social networks for fans. Each team has a list and tweets have proved to be more successful in mobilizing fans than originally anticipated. Here is a useful post by Mashable on how to create Twitter lists.

We are still very much in the experimental mode with our Titter Lists and they are all in the beginning phase. They will grow. Here are our lists. You can check them out at @darwineco or just click on the links below. We welcome suggested additions.

http://twitter.com/darwineco/enterprise-2-0

http://twitter.com/darwineco/web-2-0

http://twitter.com/darwineco/french-speaking-community

http://twitter.com/darwineco/mathematics

http://twitter.com/darwineco/km

http://twitter.com/darwineco/media

http://twitter.com/darwineco/vip

http://twitter.com/darwineco/sme

http://twitter.com/darwineco/partners

http://twitter.com/darwineco/darwin-team

My KM World and Enterprise Search Summit Notes: Evolve From a Tactical E-Discovery Approach to Search and E-Discovery

This is another in a series of notes from the 2009 Enterprise Search Summit and the 2009 KM World. This is part of the Enterprise Search Summit. It is titled: Evolve From a Tactical E-Discovery Approach to Search and E-Discovery by Brian W. Hill, Senior Analyst, Forrester Research, Here is the session description.

During the next 18 months, initiatives to cut costs and rising regulatory demands will drive enterprise e-discovery investments. Although search is a critical component of e-discovery, enterprises Forrester surveyed use a grab bag of search tools and techniques for litigation or regulatory inquiries. Lacking an end-to-end approach to gather and filter information, few enterprises report having a holistic approach to e-discovery. A full two-thirds of Forrester respondents consider their e-discovery strategy reactive rather than proactive. This indicates overall enterprise readiness for e-discovery is dismal. In this session, Hill will demonstrate the need for information and knowledge management (I&KM) professionals to collaborate with IT, legal, and business teams to understand how search technology can help cut e-discovery costs and mitigate risk. Formalized, cross-functional e-discovery programs that facilitate internal communication, encourage standardized methodologies, and provide a “big picture” view will help drive results. These programs can help support initiatives to synchronize e-discovery,
records management, and archiving.

Brian started by saying that eDiscovery solutions are not cheap but not using them can be much more expensive. Companies are grappling with regulatory issues and getting ready for possible litigation. Organizations have responded to this by rolling out a series of archives. The result is content and technology chaos that is difficult to search across and does not manage to mitigate legal risk. 

There is massive fragmentation within many enterprises. Security and metadata can be very different and not connected.

eDiscovery is a large market but it is also immature. May organizations are having challenges here. There are different stakeholders including legal and regulatory and large set of technology options. eDiscovery now invites fear uncertainty and doubt. At the same time organizations are trying to take out cost.  (see my posts: Equivio Releases Relevance™ to Enhance eDiscovery and Darwin Ecosystem Brings Awareness Engine™ to the Web and within the Enterprise)

There are different content types: email, collaboration systems, desktop files, etc. Email is the fastest growing content type.  However, the law says that all types of electronic content is subject to eDiscovery, including things like voice mail.  Only 20% of organizations surveyed felt their records were accessible and searchable.

There is a change point at the 5,000 employee level. There is more pain above 5,000 employees. Companies at this level feel more vulnerable and have much more complexity to deal with.  There is more to it than search. There needs to be proper legal hold capability within records management.   Brian looked at the records management vendors. Many people are using Sharepoint but it is not designed for this task. (see Forrester ECM wave report)

Less than half of stakeholders are satisfied with their records management.   Technology is not the answer. You need a well defined process that has proper stakeholder alignment. The market is fragmented and there are about 500 venders that address this need in some way.  But again the right process is the main need and it is perceived as the top challenge.  Defining defensible, repeatable process is key.

Records managers report all over the place in organizations. This is another example of fragmentation. There is gradual trend toward reporting form IT and legal.  IT and legal are key players here and Brian said he often finds that IT and legal have not met before.

There is a wide variety of maturity of this function within large organizations for those just starting to those very well developed. The company’s legal risk profile is a main driver of maturity, as well as how many times they have been sued.

One search approach is key words but you only find what you are looking for. You might not know the right key words.  I will add that this is a place where Darwin Ecosystem can help as it allows for discovery of relationships that you were not necessarily looking for. There are tools that use fuzzy logic, clustering, and other means to uncover key words that you did not think about.

The most commonly used technology to support search is operating system and desktop search. There are a number of shortcomings with this approach.  A very low percentage are using enterprise search.

In summary, it is much more than technology.  You need to define objectives, build queries, measure success, automate, review and analyze. You need a team of experts: legal, search, statistics, etc.   Make sure you have the right exec support. Get the right stakeholders. Develop a records retention strategy and policies. Create a consistent process.  Need to have integration between records management and search.  Should stay on top evolving case law.

Brian's slides are available at http://www.forrester.com/esswest09 

Introducing the Darwin Discovery Engine Blog

I want to share with you a new blog that I am participating in, the Darwin Discovery Engine Blog. I have been working with Darwin Ecosystem for several years in an advisory role as we work in relative quiet to get to the Alpha release phase. In August we we begin to be more public and launched this blog. I hope you will visit it and become a subscriber. 

Through this blog we want to encourage discussion about gaining greater understanding of what is occurring in the world and within the enterprise through looking at the patterns occurring in Web-based content. The Web is constantly generating vast amounts of new content and Web 2.0 adds layers of useful social information. Likewise, the introduction of Web 2.0 social media into the enterprise creates large amounts of potentially useful information about the social side of business processes. The key is gaining awareness and access to this social intelligence. Here are some sample commentary posts, Webcom 09 - The Misplaced Fear of the Mainstream Media, Wall Street Journal on Building and Monitoring Online Communities, and A New Age of Communication-Centric Collaboration. 

We assume that the content overload on the Web, and even within the enterprise, has grown to chaos so we do not use metrics such as popularity to find useful content. Instead we use correlation metrics based on Chaos Theory to unlock the relationships between content on the web. We have woven the work of Edward Lorenz into our logo through the mathematical representation of the Chaos' theory attractor. Among other things it makes spam to gain a recognition advantage virtually impossible. Our awareness engine gives users the ability to perceive and be conscious of events and patterns of activities captured in the enterprise and Web 2.0 (see our Darwin Ecosystem web site). 

Our company is bi-lingual and we post in French as well as English. Here are some recent posts written in French: Prendre le Pouls 2.0 du succes and Peut-on tirer du sens des micro-blogs? We are also on Twitter at @darwineco and we post our favorite tweets on the blog (see for example, Our Favorite Tweets for October 1 - 14 2009 from @darwineco). 

While we often cover our efforts at Darwin, this blog is open to the broader discussion of all aspects of gaining insight on Web-based content and how it can change an organization. We will also offer our perspectives on industry events and market news. Our goal is to help build a more transparent enterprise and obtain greater value from the Web, changing both how people work and the way they can better work together. We want to hear from you and welcome your comments, stories, and questions. We encourage you to browse our categories and not just look at our latest content. The categories contain the archives of our past posts. We hope that many of these posts remain useful as we will discuss issues and not simply provide current news. Also use our search field. 

Often good blog content gets buried as it moves off the front You can find the profiles of our bloggers in the left side column. They include Thierry Hubert - Founder & President, Frederic Deriot - Founder & CEO, myself, Bill Ives, VP Social Media, and Peter Mularien - Senior Architect. Links to our twitter feeds are also there. If you like what you find here please join the conversation through our comment fields. Our blog will allow us to learn of your comments regardless of when the original post was published. We look forward to hearing from your perspectives. We want to learn from you and we look forward to your participation.

Darwin Ecosystem’s Awareness Engine™ Provides Enhanced eDiscovery within the Enterprise 2.0 and on the Web

The introduction of Web 2.0 social media into the enterprise creates large amounts of potentially useful information about the social side of business processes. Enterprise 2.0 also provides a wealth of social “data” as a byproduct of its use. We are only beginning to figure out how to harvest this content. The organizations that do this are truly working in the enterprise 2.0 space.

In this post I want to bring to your attention to a new tool that I have been involved with for the past two years. We have been operating somewhat quietly and are now making our work public with both an enterprise edition and a Web edition. The Alpha version of the Web addition is available on our Darwin Ecosystem site through free registration for your use. The enterprise version is also available to early adopters.

Increasing awareness in context and finding related content is the goal of Darwin Ecosystem. Its awareness engine™ gives users the ability to perceive and be conscious of events and patterns of activities captured in the enterprise and Web 2.0. The introduction of Web 2.0 tools into the enterprise also opens up possibilities for even more information silos if these tools do not connect. Darwin provides a way to see emerging patterns of related content across information generated through a variety of tools.

Darwin reduces the effort of keeping-up with enterprise 2.0 content (Information Overload Management). It classifies and correlates patterns of the business activities trapped in your Web. Through its Scan Cloud™ Darwin makes visible and measurable the value of the enterprise 2.0 content (Awareness and Monitoring). You can see related items in a tag cloud like visualization and this relationship shifts as you move through the cloud. This dynamic nature makes it easy for users to see the correlation across the enterprise 2.0 content (Discovering and Sharing).

Using correlation metrics based on Chaos Theory Darwin looks for the emergence of correlated themes within chaotic content. This moves Darwin away from the voting and popularity rankings used by many information aggregators and search engines to rank information. By using correlation metrics, it does not require a known process or taxonomy to discover useful and related information. Darwin allows for the emergence that Paula discussed. You can set up ongoing filters or “attractors” to explore emerging themes in specific topics of interest or simply look at the broad picture.

Darwin is a not a replacement of existing enterprise 2.0 technologies, dashboards, and document management tools. It is designed to complement and leverage these technologies by making their tacit knowledge more visible. Darwin looks at events, blogs and other Web 2.0 sources that may correlate with the enterprise actions, its competitors and the voice of its critics and customers. This allows the enterprise to be capable of more easily discovering its own knowledge assets as well as its market and competitive positioning.

For example, Using Darwin’s Awareness Engine, a program manager in Marketing becomes aware of concerns in R&D, Sales or other initiatives that are emerging as he/she tries to promote a product (all without having to wait and depend the next meeting or coffee break encounter). They can also see competitive moves that may correlate with their own plans or the efforts of their customers and prospects. Likewise management can have a high-level view of what topics are emerging across all divisions and initiatives to better steer the business or measure the effectiveness of the vision’s execution, in the process discovering emerging and stronger initiatives or employees that are noteworthy.

It is a Web browser application (Scan Cloud™) or it can become a custom solution through API access. It is delivered through a Web server with services and a database correlating the different Web 2.0 sources. For the enterprise there is an on-premise solution running on Ruby on Rails and making use of RSS feeds. Its Virtual Cortex™ database can be set on Oracle, MS-SQL or mySQL according to scalability needs. You can go from inside out so an enterprise can correlate their initiatives, knowledge and business intelligence against the pulse of the Web. The application is currently in Alpha stage and available for use.

Here is an example of Darwin in action. Using the Web edition. I set up a query on social media and saw the word – gain - in the cluster shown below. I looked to see why the word - gain - is associated with social media by selecting it and highlighting the cluster of correlations between social media and gain.

Then I saw three articles that matched this correction in the past few days as you see in the image below.

There was as story about how NASDAQ launches social network site as shown below. This was new news and very interesting to me.

I also saw an article that reported in the UK that social networking sites accounts for 25% of display ads. I had read elsewhere that in the UK online revenue has passed television ad revenue so this added further context to that story. There was also an article on using Facebook traffic to drive brand loyalty. None of these articles appeared on a Google search on social media and gain. Nor did they appear on Google News on the topic. Below is a screen shot of the complete Darwin interface so you can the relationship of the detailed components shown above. This exercise is a very brief sample of how you can discover new information that is often unanticipated through correlation that is hard, if not impossible, to find in traditional search.

You can also find stories about themes that are emerging in the Web, as well as images that correlate with content as shown in results about a very recent helicopter crash in the Pacific ocean.

Darwin also allows you to correlate content in traditional mainstream media with what is happening in the blogs and other social media. In 2008 Darwin won a Young Entrepreneurs Award from the Office for Science and Technology of the French Embassy in the United States. The Young Entrepreneurs Initiative (YEi) is a platform for mentoring and networking US-based entrepreneurs who understand the importance of internationalizing their vision and their activity, and who wish to set up a technology venture in France.

We recently started the Darwin Discovery Engine Blog to provide more background on their efforts, as well as commentary on content issues on the Web and enterprise 2.0. Please join us there. Our Twitter feed is: @darwineco. We look forward to hearing from you.