You are making far too extreme a judgment here. You also say humans cannot "reliably" identify emotions from photographs. This is just setting the bar far too high. Humans can quite often identify emotions from photographs. It's far better than random chance and you shouldn't call this level of performance "snake oil".
You can criticize AI emotion detection as likely to lead to some sort of dystopia, but saying it can't possibly work is just going to lead you to an incorrect analysis of the situation.
I don't think it is setting the bar too high, "far better than random" can still be abysmal.
You're right that humans are better than random, but there is a wide range of emotions. Assume for the sake of argument there are 10 emotions to identify and evenly distributed in frequency, then random is getting it right 10% of the time. Let's say humans get it right 30% of the time from still photographs of real-life situations (not staged) -- I'm making this specific number up but depending on the type of experiment, it's a reasonable ballpark one.
That's still horribly low, wrong more than half the time. I can't think of any situation where that could warrant any kind of responsible decision-making. If we were talking 95% accuracy then there could be value, I'm not asking for 99.99% accuracy. But if reliable detection is something like 30%, I stand by calling this "snake oil" even if it's not just random.
Well Dr. John W. Thackery had too much pride to be associated with that and refused the money.
Cut to modern day version, I'm a computer programmer and I write code to correctly place humans from video feeds into various buckets based on the RGB colors I scan from bitmap image 2d arrays... and I say it places them into buckets like "possible upset" and "very much in flow state" or "bored, brain not engaged." And then sell this program/SAAS product to companies and they get value from these buckets... how is that still true snake oil?
E.g. take class A of students from one culture and class B of students from another culture, and assume the students all feel the same level of moderate anger, but the two cultures have different "anger display rules" as they're called.
The detector might judge all the students from one class as angry, but none from the other. Aggregating students into classes doesn't fix anything.
Similarly, if you took some students who have a certain expression when they're happy, and other students who have the same expression when they're afraid, and mixed them together in the same class, then whatever emotion the detector is interpreting the expression to mean is going to be triggered by completely unrelated things, and the signal won't be meaningful at all.
Yes, there are differences - especially in identification of intensity. But in general detection is pretty reliable across cultures.
Eg:
The results show that the overall accuracy of emotional expression recognition by Indian participants was high and very similar to the ratings from Dutch participants. However, there were significant cross-cultural differences in classification of emotion categories and their corresponding parameters. Indians rated certain expressions comparatively more genuine, higher in valence, and less intense in comparison to original Radboud ratings. The misclassifications/ confusion for specific emotional categories differed across the two cultures indicating subtle but significant differences between the cultures.
https://journals.plos.org/plosone/article?id=10.1371/journal...
Unfortunately that's simply not true. Your example merely shows reliabiliy between two countries, not all countries. You can research, for example, how Japanese people tend to smile when angry, sad, or embarrassed, in total contrast to Westerners.
This is one of the classic "extreme" examples, but it demonstrates how my point holds -- errors are not random but are highly correlated with the culture -- not to mention the subculture, the individual (the person who hides all emotion when angry), etc.
isn't that a pretty sane and basic assumption ?
Seriously though, especially with individuals, why shouldn't you use all the information available, if it produces results even an iota better than blind acceptance (and for most people, their empathic-sense does)?
In the context of real life, where we know people, are interacting with them, and are reading cues from their whole body and their movement over time, then we're much better at assessing emotions and we'd be pretty foolish to ignore emotional signals.
- Identifying emotions of humans who grew up in a different culture, for which the ML algorithm may have piles of data from every part of the world and you don't
- Identifying emotions of babies based on instinctual behavior
- Identifying emotions of people who are undergoing a trauma response that most humans are not trained to recognize
- Identifying emotions of people during a negotiation process in which the people are actively trying to hide their emotions from humans but nevertheless leak certain signs of their true emotions
I believe there are a lot of examples of AI doing things that humans can't. Typically it's a matter of scale, but sometimes it's a matter of some correlations being beyond the basic capabilities of humans. I could be wrong, but I'm not sure that this is the best metric for whether AI could do something.
But second, the correlations of individual facial muscle contractions with emotions has been extensively studied and it's far noisier, inconsistent, or completely devoid of a signal than many people assume. In academic terms there's no such thing as a reliable emotional "signature" to be gleaned from facial muscle activation.
So the point is, it appears that the raw data simply isn't there for the AI to detect patterns that humans can't. Detecting emotions requires far more data points outside of facial muscle activation -- such as the ones I listed.
Yes it can. You get enough labelling and it overcomes the unreliability of detection by any single human.
Get 70 people to label the same face. A random distribution over the 7 Ekman emotions[1] will give 10 each, and any non-random variation from that is a signal. Do that over enough faces and you'll get something to train on.
(Also, no reason why it needs to be face pictures. It could be 2 second video snippets for example).
[1] https://www.paulekman.com/blog/darwins-claim-universals-faci...
The fundamental problem is that multiple emotions can result in the same facial expression, and the same emotion can result in multiple facial expressions.
It doesn't matter how many people label a face. It won't get over the fundamental issue that there is no 1-1 mapping between emotions and facial expressions.
Actual experts disagree. For example Ekman, for the basic emotions at least (see the link I posted above). To quote:
In the late sixties, Izard and Ekman in separate studies each showed photographs from Tomkins’ own collection, to people in various literate cultures, Western and Non-Western. They found strong cross-cultural agreement in the labeling of those expressions. Ekman closed the loophole that observing mass media might account for cross cultural agreement by studying people in a Stone Age culture in New Guinea who had seen few if any outsiders and no media portrayals of emotion. These preliterate people also recognized the same emotions when shown the Darwin-Tomkins set. The capacity for humans in radically different cultures to label facial expressions with terms from a list of emotion terms has replicated nearly 200 hundred times.[1]
The FACS coding system[2] has pretty good experimental support, across multiple cultures. As noted on this article, it can distinguish between fake smiles and real smiles etc. See https://en.wikipedia.org/wiki/Smile#Duchenne_smile for more on this.
Even if you disagree with that and think that motion is required (which is an argument that has some validity) there is still no reason to think a computer system can't do it.
[1] https://www.paulekman.com/blog/darwins-claim-universals-faci...
[2] https://en.wikipedia.org/wiki/Facial_Action_Coding_System
Ekman was at the forefront of creating facial emotions expression research, but while I do believe he convincingly showed that there are basic instinctual, stereotypical, cross-cultural facial expressions, the field now generally accepts that FACS coding does not map reliably to emotions in real-life contexts.
In other words, just because a smile is a cross-cultural instinctual display of happiness does not mean a given individual who is happy will be smiling, or that a given individual who is smiling will be happy.
The same applies for labeling, if a particular facial expression can occur in an indistinguishable manner because of three different causes, that's okay; there's nothing wrong with a gold standard label in training data marked as "we've observed that 50% people had this expression because they were angry and 30% because they were in pain, but no happy or sleepy people had an expression like that".
Furthermore, for the purpose listed in the article, you don't need to determine an absolute value of some emotional state, but you need to detect large shifts in it (i.e. you get an baseline that implicitly adjusts for part of individual and cultural differences) - i.e. whether the audience (in aggregate!) now looks significantly more frustrated than the same people looked 30 minutes ago.
That's interesting. Where can I read about these issues?
I think this is a pretty decent example of a Neural Net becoming better at something than humans are.
What was the situation? What expressions was the person making in the previous moments. As you said, body language. How the person relates to the viewer.
A video clip would be better but still would miss a lot.
Please consult the current state of AI before giving an a priori rebuttal. Emotions can also be detected in text, voice and video, not just in static images. Sentiment detection in text is widely used.
The premise of the article is valid, whenever AI systems are being deployed they need to be ethically vetted, but the writing is superficial and 'too emotional'.
Sentiment detection in text, for example, is not emotion detection. That's not a rebuttal.
As an example of what I mean, you can feed a NYT news article written by a journalist experiencing regret, or boredom, or elation, due to events at home, while writing it -- and it won't pick up a thing. But it may pick up a valid sentiment of outrage, which was intended by the author as the desired effect upon the reader, but which was not once felt as an emotion by the author while writing it.
Semantic detection is far easier because the author of text generally tries to communicate a sentiment, consciously, in their text, and text is intended to have fairly unambiguous meaning. Neither is generally true of emotions on the face.
And nowhere did I say emotions only exist in static images. Of course they are communicated through voice, video, etc. -- I don't know how anyone could think otherwise.
Your initial post argued about it being difficult to detect emotion given just a single frame, then concluded that "any AI products designed to supposedly recognize emotions are necessarily snake oil".
And I stand by the snake oil comment for the reasons stated, with the state of AI and emotions research today.
I'm not saying it can never be done, but that we would need major breakthroughs in both.
Sentiment detection continues to be an entirely different area of research.
As a layman, I had previously thought Paul Ekman's work refuted this claim, but it seems like it's been called into question. (Ekman was reportedly the basis for the main character in the show Lie to Me)
I wish people and machines would Not try to infer too much from this sort of thing. Treat people as ends in themselves.
Unfortunately snake oil is a profitable business, and unreliability isn't going to stop people from selling and using these systems.
It's going to be like AI-driven captchas [1], the people subjected to these systems are going to have to learn to give the systems the answers they want, even if it's obvious they're incorrect. You'll have students concentrating on expressing an "attentive" face rather than concentrating on the material.
[1] Yes, I'm talking about you reCAPTCHA. Highway buzz strips aren't crosswalks.
This is absolutely not the case. I work in adjacent fields, and it not uncommon for machine learning based solutions to reliably outperform any single human.
In most cases ML aggregates human judgement. Provided some humans are right more of the time than a random distribution would give (and there is enough data) then a good learning algorithm will find it.
Now there are issues with this of course - inappropriate generalisation, inappropriate overfitting etc.
And this comment is correct - data outside a single photo will make the system much more reliable. 2 second video sequences should help a lot.
But the general principle is valid: an AI/ML should be able to reliably outperform humans.
So emotions can be thought as a language (albeit a pretty lossy one). If AI can one day understand spoken/written language, I don't see why it can't also understand the language of emotions.
The second doesn't necessarily follow from the first—there are plenty of things humans are bad at that computers are good at.
You are probably right that current products aren't very capable, and may even be snake oil.
But also I'd be quite surprised if this isn't possible in the near future.
The leading example of the article was about Zooom videos, not single frames, and in a school setting you presumably know what culture the students are from, so I don't think you need it to work cross-culturally. And I don't doubt that people can deliberately hide their emotions, but many settings are not adversarial, so there would be no reason to do so. E.g., if you imagine that a teacher is teaching a lecture to a large audience over zoom and is using a "puzzlement meter" to see if they seem to be getting it, then any audience member who does feel puzzled will only gain from frowning and making the lecturer slow down.
In general, I think from my experience talking to people over video chat, some emotional information does get communicated over the video, so in principle computer programs should be able to pick up on it too.
The products in question are almost certainly assessing still frames from video. There's been vanishingly little research on the time component of emotions, even though we know it's hugely important.
> you need it to work cross-culturally
This would then require separate training models for e.g. individual subcultures within countries as well as detecting which subcultures participants belong to. That is also far beyond anything being done currently.
> but many settings are not adversarial, so there would be no reason to do so
It has nothing to do with an adversarial setting, people try to hide their emotions constantly. They hide that they're fed up with their boss in front of colleagues, they hide that they're stressed with their spouse at work, they hide that they're worried the project will fail. We are emotionally regulating virtually all the time.
> some emotional information does get communicated over the video, so in principle computer programs should be able to pick up on it too
In principle, yes, but in practice the emotional content is so dependent upon your cultural and individual mental model of the person that you would need to model their entire psychology. "What does that long pause mean?" The point is that emotional signals are so incredibly complex and vary so much from person to person, that the difficulty of accurately decoding emotions is more akin to AI that can make conceptual inferences and hold a genuinely intelligent conversation, as opposed to mere pattern recognition.
But there was a little test that ran around a few years back that was just “Is this smile genuine or not” and I was able to identify 20 out of 20 pictures real and fake smiles.
The pictures were gathered by asking subjects to smile and then taking the picture or taking a picture after they responded to a joke, so I guess there is still room for some of the “genuine” smiles to be faked.
The issue is the emotion behind it... you can be quite happy without smiling at all, and people who pose for the camera all day long learn to give a 100% convincing Duchenne smile no matter how they're feeling that day.
i.e. you know you got a rush, then by examining context, you attach a (theory-theory) word to that rush. Both AI and photographs are bad at context.
You say that like human intelligence is some sort of upper bound on intelligence.
And the fact is that humans can't reliably identify emotions from photographs, so there's zero chance for AI.
While I don't agree with much of what she writes, the psychologist Lisa Feldman Barrett gives an excellent overview of why at the start of her book "How Emotions are Made".
But a short explanation is that 1) emotions are expressed not just with the face but with the whole body, 2) the expression of emotion is a sequence of changes over time that can't be captured in a single frame, 3) there is extreme overlap in the expression of emotions (e.g. a seemingly grimaced look could either be painful anguish or ecstatic surprise) and 4) while we do have instinctual expressions for emotions (smile/frown/etc.), we learn culturally how and when to hide, show, fake, or otherwise modify these by habit, as well as consciously do so for our immediate purposes.
So we humans obviously do recognize emotions in others, but we do so with a huge variety of cues, we still frequently get it wrong, and we do a far better job when we share the same culture, and also know the individual person well. (Think how some people express anger with aggressive shouting, others by silently withdrawing.)
Most evidence for the idea that emotions can be identified (and cross-culturally) comes from the research of Paul Ekman, but this was limited to 6 "basic" emotions with photographs of highly exaggerated/stylized faces performed by actors. It has no relevance to whether real-life, non-"staged" emotions can be reliably detected.
So any AI products designed to supposedly recognize emotions are necessarily snake oil. I assume they are reliably measuring facial expressions, but facial expressions simply cannot be mapped to actual emotions in any kind of reliable way in the real world.