Re:Sentiment Analysis on Enron's Emails with Apache Spark
by mindcrime on Friday May 06, 2016 @11:09PM
(#11648215)
If this type of analysis can't answer "who Ken Lay likes best", maybe that's not the best question to use as an example.
"Who did Ken Lay like best" was originally going to be the title of the post, until we realized that you can't actually answer that question. It was someone else's idea anyway, but I'm not naming names!
That said, I only had so much time to work on this, and there was a limit to how much depth I could go into. I wanted to go deeper and introduce some network science / social network analysis but had other priorities.
But still, the point was just to illustrate a cool application of that particular technology stack and pique people's interest in it. If that interest helps us land business, that's a nice outcome. If not, maybe somebody learned something or came up with an idea for their own project, as a result of reading it. Those are nice outcomes as well. :-)
There are plenty of more interesting questions ("when did Enron employees start getting nervous or alarmed?")
That is a pretty good one... maybe somebody else will take the bull by the horns and write a post on that. If not, maybe I'll do it later for a "part 2".
Re:Sentiment Analysis on Enron's Emails with Apache Spark
by mindcrime on Friday May 06, 2016 @11:12PM
(#11648225)
LOL, no. It's just a fun application of some neat technology. We like to publish those kinds of things on our company blog. It's a chance for our staff to do something fun in between engagements, it helps promote the company, and somebody may learn something from it. We see that as a win-win.
There are plenty of more interesting questions ("when did Enron employees start getting nervous or alarmed?") that can be answered by this dataset.
"Who did Ken Lay like best" was originally going to be the title of the post, until we realized that you can't actually answer that question. It was someone else's idea anyway, but I'm not naming names!
That said, I only had so much time to work on this, and there was a limit to how much depth I could go into. I wanted to go deeper and introduce some network science / social network analysis but had other priorities.
But still, the point was just to illustrate a cool application of that particular technology stack and pique people's interest in it. If that interest helps us land business, that's a nice outcome. If not, maybe somebody learned something or came up with an idea for their own project, as a result of reading it. Those are nice outcomes as well. :-)
There are plenty of more interesting questions ("when did Enron employees start getting nervous or alarmed?")
That is a pretty good one... maybe somebody else will take the bull by the horns and write a post on that. If not, maybe I'll do it later for a "part 2".
More like a, you know, "Look, Ma -- I got Spark, Hive, and HDFS on my resume now!" analysis.