Clickstream

Natural Language Processing, Social Networks and Enron Emails

This article by Jeffrey Heer at UC Berkeley describes an application called enronic he constructed to analyze the content of Enron's corporate email. He's making use of a several different visualization techniques and algorithms for message categorization, social network inference and community analysis for extracting information from email text.

His reasoning behind doing this work is exactly the right way to attack the seemingly intractable problems with text mining and natural language processing, as well as the slow adoption of data mining technology. Instead of working on improvements in the algorithms to make them better or more usable, he focuses instead on combining visualization techniques with the underlying algorithms to allow easy exploration. The goal is to enable a person to explore data and relationships instead of trying to replace the person with a system that spits out the answer.

Here he talks about a discovery he made while exploring the data with his application:

Such an analysis revealed the role of John Shelk, who regularly reported on Congressional meetings, sending all such meeting reports to Tim Belden. In fact, the visualization reveals that their conversation is completely one-sided, with John sending reports to Tim, with no back-traffic occurring. This is a bit suspicious. Clicking on Tim Belden then reveals that according to the database he hasn't sent ANY e-mails, but receives various legal reports from throughout the company.

This is an interesting occurrence to flag for increased investigation. It could be that Tim Belden's e-mails simply were not included in the annotation set given to the class, and this would be easy to check out (of course, ideally the interface would support this check already, a point for future improvement). Otherwise, might have his e-mails been expunged? If so why? It is clear that Tim Belden was receiving important information from throughout the organization -- what happens to this information subsequently? Is it being utilized outside of e-mail? This might highlight Tim Belden as a possibly interesting person for prosecutors investigating Enron.

After performing this analysis, I did a web search on Google for "Tim Belden". On my honor, I had never heard his name before doing this analysis exercise. Little did I know he was the first person charged by prosecutors, considered the "mastermind" of Enron's manipulation of California's markets, and was found guilty on charges of federal conspiracy. The full story is available at CBS News. Already, this bodes well for the potential of the system!

Posted by Mark Wednesday, June 08, 2005 9:16:00 AM | permalink |