I find it hard to see how detecting and eliminating sarcasm requires a theory of mind. It requires some association between various stylistic elements and the concept of sarcasm.
The same is true of irony.
I still wonder how many of these people have read Dennett's "The Intentional Stance", which holds that the best way to think about "intention" is as an explanatory model, not a mechanism. That is, we can say that the dog "behaves as if it has the intention to get inside" without making any claim about the internal state of the dog.
Dennett further speculates that our own self-experience of intention is a matter of turning the same explanatory model upon our own behavior, but that's an extension that isn't directly relevant to this speculation about language models.
That's also a way to frame evolution and literally all of biology.
That sounds very mechanic-ist, including Dennett's theory (I haven't read Dennett, to be exact, I'm going by your explanation). Which I guess it's par for the course when talking about a scaled up Mechanical Turk concoction like this GPT thing is. It won't "create" any Radio Erevan jokes anytime soon, that's for sure, though.
I am quite certain that detecting sarcasm can't be done based on stylistic elements" and requires some (even if implicit) estimation of what the author is thinking that contrasts with what is being said.
E.g. a relevant example I have actually seen in doing sentiment analysis on tweets to evaluate how customers perceive a company is "#CompanyName Got my order delivered in just under three hours. Thank you for great service! thumbsup-emoji" - now is this a positive review or sarcasm? And the thing is, you can't tell from the message by itself, you need an understanding of the customers' expectation (guess you might call it "a theory of mind") that for a pizza chain this is obviously sarcasm, but for a web store that sells some electronics the same thing would actually mean a fast delivery and great service. IMHO sarcasm detection is mostly about 'world knowledge' about what the implied expectations are, and not about stylistic elements at all.
If you have determined that a common element in reviews is a reference to delivery of goods or services within a given timeframe, and can identify that most of the quantitative descriptions of the timeframe are relatively short ... then coming across one that uses a much larger timeframe but is still positive will be quite noticeable.
This is actually typical of the problem with far, far too many people's interpretation of what language models are doing. The process I've described requires only a representation of language behavior. The LM can be said to understand how people talk (write) about a thing, but there is no knowledge of anything beyond language behavior.
Though, with LLMs, you might need quite a few (often more than 5) exchanges to find out that it can't actually quack.
The first notable point is that the model caught that (on its first read, FWIW), even though the original human doing the labeling didn't.
And the second is that the researchers discovered this, and presumably discussed it. And yet when they wrote up the paper they not only dropped the content but denied that the analysis had been done.
> I am so happy not to live in an American small town. Because whenever I'm shown some small town in the States it is populated with all kinds of monsters among whom flesh hungry zombies, evil aliens and sinister ghosts are most harmless.
Mocking irony, in the context of a negative review.
What I find fascinating here is how generative AI has inverted all our sci-fi tropes. I mean, sure: you're right! That's the way "sarcasm" is defined in most dictionaries. But you and I both know that as the language is actually used, the term means a whole host of techniques used to convey negative emotional content in language that is not directly negative. Your (correct!) dictionary pedantry isn't interesting to me. We've been here before.
But GPT-4 wasn't trained on dictionary rules. It was trained on actual language. And it's actually better at inferring this stuff than the pedants are. Our introvert brains have trouble teasing meaning like this and have to hide behind rules and structure. The computer doesn't.
To wit: the robot is us.
There is another message that the reviewer doesn't like the genre, but that isn't a comment on the movie.
There is no way a human reading this review would think its positive.
"I'll just tell you that it's slightly reminiscent of U Turn by Oliver Stone but is a way down in all artistic properties."
Oh and UTurn only scores 6.7 so its a bit of a crap movie to start with.
So I guess maybe it’s saying the human was wrong after all?
Maybe more like appreciation with some humour thrown in?
Are we all a bit confused and second guessing our own use and interpretation of language now ?
That's probably the least sarcastic bit for me, because it's just a reference back to the opening sentence which says: "I'm glad I don't live in a small american town because bad things happen".
Anyway, I think the conclusion is that it's pretty hard to tell and if our new robot overlord says it's sarcastic, I guess we better listen...
He gave it a score of 7 which IMO is pretty high?
And yes it absolutely is subjective. That’s exactly the point and the power of these LLMs, to be able to handle the vagueness of human communication.
"we asked DV3 for an explanation. DV3 replied that it detected sarcasm in the review, which it interpreted as a sign of negative sentiment. This was a surprising and reasonable explanation, since sarcasm is a subtle and subjective form of expression that can often elude human comprehension as well. However, it also revealed that DV3 had a more sensitive threshold for sarcasm detection than the human annotator, or than we expected -- thereby leading to the misspecification.
To verify this explanation, we needed to rewrite the review to eliminate any sarcasm and see if DV3 would revise its prediction. We asked DV3 to rewrite the review to remove sarcasm based on its explanation. When we presented this new review to DV3 in a new prompt, it correctly classified it as positive sentiment, confirming that sarcasm was the cause of the specification error."
The published paper instead says "we did not test for the ability to understand sarcasm, irony, humor, or deception, which are also related to theory of mind" .
The main conclusion I took away from this is "the remarkable emergence of what seems to be increasing functional explainability with increasing model scale". I can see the reasoning for why OpenAI decided not to publish any more details about the size or steps to reproduce their model. I assumed we would need a much bigger model to see these level of "human" understanding from LLMs. I can respect Meta, Google, and OpenAI's decision, but I hope this accelerates the research into truly open source models. Interacting with these models shouldn't be locked behind corporate doors.