But what tests have you done on your broader methodology? What experiments can you really do to figure out the extent to which the use of the word "nosegay" is correlated with actual depression?
Also, as someone else said, where are the error bars? If there really is a correlation between word choice and other metrics, then some simple statistics should give you error bars on your other metrics, right?
Oh, one more thing: in the example on your website you say that the sentence:
"That joke kills me!"
is "subconsciously" aggressive. My question: would your algorithm rate that at exactly the same level of aggression as the sentence:
"I'm gonna kill you!"?
cuz, y'know, intuitively one seems rather more aggressive than the other.
Please see my response to the original question about this. [1]
"That joke kills me!" vs. "I'm going to kill you!" (corrected grammar as we do not usually focus on slang)
They do not score exactly the same, though they are both very high on anxiety, hostility, and depression. The former has high marks for happiness and compassion due to the word "joke" being used.
For example, words that are generally positive may be only used in a snarky context in hacker circles, whereas words that are generally negative may similarly be used with positive affect (e.g. the word "hacker" itself). Did you customise your lexicon at all for this genre?
Also, in my experience, it's not the lexicon of words -- heck, SentiWordNet has existed for long enough -- it's what you do with that input signal that counts. Many "bag of words" approaches are on the shelf now, and very few of them are particularly accurate or clever, but most of them do the job just well enough that they get away with it.
That's a great insight and I believe you are correct. In general, I think the domain does have an impact and I am working on some automatic demographic profiling technology. However, for a first-look, I think our general word dictionary covers an abundant set of terms that do not vary much between domains. For instance, your comment has the following anxiety-evoking words:
not, important, but, example, positive, used, hacker, negative, long, bag, but, away
Of all those words, only "hacker" would likely need to be adjusted for the HN domain. For a first look, we'll chalk that up to some minor added noise in the data.
EffectCheck does include the ability to automatically profile the document type (e.g., "Press Release" or "Motivational Speech") based on a training corpus. For example, most press releases have a high degree of confidence-- based on a corpus of press releases, we normalize the levels so that "Very High" on confidence for a press release requires much higher confidence elicited per-word than it would in a complaint letter.
Anyway, I guess I'm not clearly understanding your definition of "anxiety" because few of those words evoke anxiety as I would define it -- not? bag? away? I'll try to dig into whatever you've published to deeper understand your system, but I'm not convinced by the standalone word approach yet, though I see what you're trying to do (I think). I'd really want to check out the bigrams/trigrams, even if you don't do full parsing or any intermediate sophistication. Obviously there's an efficiency aspect, so I guess if your results are good enough already and you have customers, no need to make sacrifices.
On a tangent, how does your system interpret common HN words like "healthcare", "Obama" and "education", I wonder? Could you have an "intellectual" or "hacker-ish" dimension based on manual categorisation of words into "this is stuff HN should be talking about" vs "this is stuff that belongs on reddit"?
Actually, both. Corporate marketing departments spend a lot of effort crafting the proper press releases. Entire agencies exist to help companies with press releases. I think we make a great companion tool for professionals in that area. We have an example of how EffectCheck can help with press releases on our blog [1].
>On a tangent, how does your system interpret common HN words like "healthcare", "Obama" and "education", I wonder? Could you have an "intellectual" or "hacker-ish" dimension based on manual categorisation of words into "this is stuff HN should be talking about" vs "this is stuff that belongs on reddit"?
We certainly can do that, and we offer it as a service to clients that want hand-crafted, demographic-specific dictionaries (like political campaigns).
[1] http://blog.effectcheck.com/2011/04/14/delivering-bad-news-a...
I'm a little skeptical.
It rates "anxiety" as "medium-high." Yet I don't feel the slightest bit anxious reading that page. Nor does the author come off as anxious. It all feels very positive to me. (I'm counting the text alone, not the graphic design.)
It rates "compassion" as "very high." I wasn't able to find a description of what exactly the compassion measurement means in EffectCheck. But I'll go out on a limb and assume a high compassion rating means a text will either make you feel like the author is compassionate, or make you feel compassionate yourself. I don't feel either for that page. Not that it's hostile--I just don't feel anything on that axis when I read the page.
It's entirely possible that I've misunderstood how EffectCheck's ratings should be interpreted. It's also possible that I'm missing the point by comparing how I think I feel against EffectCheck's results. Maybe we're unreliable observers of how a text emotionally affects us, and EffectCheck more accurately predicts what's really going on deep in our brains. In which case, forgive my unwarranted naysaying.
Bingo. Our algorithm focuses on the lingering, subliminal emotions that you aren't necessarily aware you're feeling. Though not related to the emotions we measure, one commonly-referenced study shows that if you're walking down the street and you see a big sign with the word "ELDERLY" written on it, you'll actually walk slower without realizing it [1]. Lots of words have these subconscious impacts and you simply cannot accurately poll your own brain to determine how you feel.
[1] See "Stumbling on Happiness" for a great overview of this study: http://www.amazon.com/gp/product/1400077427/ref=as_li_ss_tl?...
The remaining question for me is whether and how EffectCheck's results have been compared against actual measurements of these subliminal effects.
So there seems to be a lot of skepticism about whether our algorithm can actually measure emotional impact accurately. For the long answer, I'll refer you to the about page[1] for EffectCheck. For the short answer:
My co-founders are an AI PhD and a Clinical Psychology PhD. They spent three years curating a huge dictionary of words using a methodology similar to the Harvard Psychosocial Dictionary [2], but with the twist that they were focused on lexical impact of words. The dictionary is pretty accurate at measuring both impact and sentiment [3]. For example, we can predict Amazon reviews as being positive or negative, using the stars to validate if we are correct-- blog post on that coming soon.
Regarding context of the word usage: Both behavioral studies and fMRI scans have confirmed that context is not as important as one might believe. Our brains process multiple meanings of words in parallel, and the emotions associated with those words linger in our subconscious even after we know the correct context. Similarly, in cases of reviews and comments, people who use hostility-evoking words are often hostile themselves (angry people tend to make others angry) and the same is true of the other five fundamental emotions we measure.
Happy to answer more questions. Also happy to analyze any data that you would like to see in order to verify accuracy of the algorithm-- just give me a link or the text. :)
[1] http://effectcheck.com/about
[2] http://www.wjh.harvard.edu/~inquirer/homecat.htm
[3] Note that sentiment is correlated to impact only if the writer or speaker is writing without detailed attention to word choice. For example, political speech writers comb over every word to make sure they have the desired impact-- thus, the final text likely has little correlation to the sentiment of the orator.