News for hackers. Stuff that matters.
Re:Microsoft's paper on OpenAI's GPT-4 had hidden information by meghan_rain on Thursday March 23, 2023 @09:01PM (#35283709)
Wait so they tested if it detects sarcasm, it failed, and then they were like "let's pretend this never happened" and wrote "We did not test for the ability to understand sarcasm"?
Re:Microsoft's paper on OpenAI's GPT-4 had hidden information by knaik94 on Thursday March 23, 2023 @09:19PM (#35283861)
I think you might have a misunderstanding. The "error" was misclassifying the sentiment of this imdb review [1]. The researchers concluded that the model was more sensitive to sarcasm than the human reviewer.

https://www.imdb.com/review/rw1756778/

Re:Microsoft's paper on OpenAI's GPT-4 had hidden information by m3kw9 on Thursday March 23, 2023 @10:18PM (#35284255)
It’s a poorly written review and anyone reading wouldn’t find it useful. Note that 1/6 found it useful in the site
Re:Microsoft's paper on OpenAI's GPT-4 had hidden information by ajross on Thursday March 23, 2023 @10:50PM (#35284448)
It's not that poorly written. But regardless: it's clearly a poorly written, sarcastic, negative review. Read it a second time and you'll pick out the sarcasm for sure. The reviewer thought this was a dumb film.

The first notable point is that the model caught that (on its first read, FWIW), even though the original human doing the labeling didn't.

And the second is that the researchers discovered this, and presumably discussed it. And yet when they wrote up the paper they not only dropped the content but denied that the analysis had been done.

Re:Microsoft's paper on OpenAI's GPT-4 had hidden information by gleenn on Thursday March 23, 2023 @11:36PM (#35284778)
Can you please specify at least a part or two that you specifically feel are sarcastic? The author doesn't seem to like the movie but I feel like the majority of the comments are quite factual or seem like straightforward opinion.
Re:Microsoft's paper on OpenAI's GPT-4 had hidden information by knaik94 on Thursday March 23, 2023 @11:42PM (#35284833)
The part of the review where it says "he must have never in his life seen a flick about any small towns" could be classified as sarcastic. The reviewer also says "you should watch the movie" if you are curious about the ending, but the overall intent of the review seems to be the opposite.
Re:Microsoft's paper on OpenAI's GPT-4 had hidden information by hn_go_brrrrr on Friday March 24, 2023 @12:15AM (#35285013)
I would call that acerbic. There's none of the reversal in sense I expect to see in sarcasm.
Re:Microsoft's paper on OpenAI's GPT-4 had hidden information by sdenton4 on Friday March 24, 2023 @02:24AM (#35285676)
in 2023 the goalposts on agi have moved far enough that we are in the bleachers arguing about whether a model was correct to classify a movie review as sarcastic, when it may have merely been acerbic.
Re:Microsoft's paper on OpenAI's GPT-4 had hidden information by deadpannini on Friday March 24, 2023 @07:19AM (#35287560)
The opening qualifies as sarcastic:

> I am so happy not to live in an American small town. Because whenever I'm shown some small town in the States it is populated with all kinds of monsters among whom flesh hungry zombies, evil aliens and sinister ghosts are most harmless.

Mocking irony, in the context of a negative review.

Re:Microsoft's paper on OpenAI's GPT-4 had hidden information by gleenn on Friday March 24, 2023 @11:55AM (#35290850)
Sarcasm is saying one thing and meaning the opposite. If monsters are most harmless, then also not wanting to live there makes logical sense and isn't backwards. So that's straightforward mockery, not sarcastic.
Re:Microsoft's paper on OpenAI's GPT-4 had hidden information by ajross on Friday March 24, 2023 @12:09PM (#35291134)
> Sarcasm is saying one thing and meaning the opposite.

What I find fascinating here is how generative AI has inverted all our sci-fi tropes. I mean, sure: you're right! That's the way "sarcasm" is defined in most dictionaries. But you and I both know that as the language is actually used, the term means a whole host of techniques used to convey negative emotional content in language that is not directly negative. Your (correct!) dictionary pedantry isn't interesting to me. We've been here before.

But GPT-4 wasn't trained on dictionary rules. It was trained on actual language. And it's actually better at inferring this stuff than the pedants are. Our introvert brains have trouble teasing meaning like this and have to hide behind rules and structure. The computer doesn't.

To wit: the robot is us.

Re:Microsoft's paper on OpenAI's GPT-4 had hidden information by gleenn on Monday March 27, 2023 @04:02AM (#35324094)
But I'm not being a pedant, we're measuring sarcasm. Someone has to clearly define it to judge the AI. If everyone in these threads thinks sarcasm is anything "negative" then they are wrong. That would be a negative sentiment classification. You have to be clear with what you're measuring.
Re:Microsoft's paper on OpenAI's GPT-4 had hidden information by thaumasiotes on Friday March 24, 2023 @03:57AM (#35286267)
It's not clear to me that it's a negative review. The movie is rated as "artistically worse than a movie by Oliver Stone". Other than that, the message appears to be that the movie is typical of its genre, which is not generally considered a bad thing.

There is another message that the reviewer doesn't like the genre, but that isn't a comment on the movie.

Re:Microsoft's paper on OpenAI's GPT-4 had hidden information by jay_kyburz on Friday March 24, 2023 @12:23AM (#35285066)
The last sentence, the conclusion, says it's like some other move, but worse. Way worse.

There is no way a human reading this review would think its positive.

"I'll just tell you that it's slightly reminiscent of U Turn by Oliver Stone but is a way down in all artistic properties."

Oh and UTurn only scores 6.7 so its a bit of a crap movie to start with.

Re:Microsoft's paper on OpenAI's GPT-4 had hidden information by salad-tycoon on Thursday March 23, 2023 @09:17PM (#35283850)
There was a conflict of opinions with D3 and human annotator. The quote above at least does not indicate who was right or wrong, merely noting that the machine had a more sensitive sarcasm detector.

So I guess maybe it’s saying the human was wrong after all?

Re:Microsoft's paper on OpenAI's GPT-4 had hidden information by ChatGTP on Thursday March 23, 2023 @10:07PM (#35284160)
If you read the IMDB review, there is no sarcasm in it.

Maybe more like appreciation with some humour thrown in?

Are we all a bit confused and second guessing our own use and interpretation of language now ?

Re:Microsoft's paper on OpenAI's GPT-4 had hidden information by knaik94 on Thursday March 23, 2023 @10:55PM (#35284482)
I got confused about whether the review was positive or negative too. The part of the review where it says "he must have never in his life seen a flick about any small towns" could be classified as sarcastic. The review isn't very clear in terms of prose and conclusion. The reviewer clearly says "you should watch the movie", but after re-reading it a few times, I would consider that as sarcasm too. I can't say for sure whether the reviewer enjoyed it themselves.
Re:Microsoft's paper on OpenAI's GPT-4 had hidden information by ChatGTP on Friday March 24, 2023 @05:49AM (#35286986)
"he must have never in his life seen a flick about any small towns"

That's probably the least sarcastic bit for me, because it's just a reference back to the opening sentence which says: "I'm glad I don't live in a small american town because bad things happen".

Anyway, I think the conclusion is that it's pretty hard to tell and if our new robot overlord says it's sarcastic, I guess we better listen...

Re:Microsoft's paper on OpenAI's GPT-4 had hidden information by dougmwne on Thursday March 23, 2023 @10:56PM (#35284494)
It sounded jaded and a little sarcastic to me. I don’t think the reviewer liked the movie much, found it all that original or entertaining.
Re:Microsoft's paper on OpenAI's GPT-4 had hidden information by ChatGTP on Thursday March 23, 2023 @11:32PM (#35284760)
Then it’s quite subjective then , who’s right ?

He gave it a score of 7 which IMO is pretty high?

Re:Microsoft's paper on OpenAI's GPT-4 had hidden information by knaik94 on Thursday March 23, 2023 @11:47PM (#35284872)
Without the context of how the reviewer rates other movies, it's not possible to say whether 7 is high or low. I would say 7 is mid or mediocre, something I wouldn't go out of my way to watch. The trend I've noticed is that most people only use the top half of the scale, anything 5 and less is bad. The funny thing, and I am guilty of this myself, is when they only use 6 through 10 but then use decimal points too.
Re:Microsoft's paper on OpenAI's GPT-4 had hidden information by dougmwne on Friday March 24, 2023 @09:47AM (#35289025)
I doubt the score would have been part of sentiment analysis. I ignored it and tried to make a judgement based on the text alone. It seemed more like a 5/10 review to me and mildly negative.

And yes it absolutely is subjective. That’s exactly the point and the power of these LLMs, to be able to handle the vagueness of human communication.

Re:Microsoft's paper on OpenAI's GPT-4 had hidden information by og_kalu on Thursday March 23, 2023 @09:17PM (#35283844)
It didn't fail ?