GPT-4o's Memory Breakthrough (Needle in a Needlestack)

Re:GPT-4o's Memory Breakthrough (Needle in a Needlestack) by Aeolun on Tuesday May 14, 2024 @08:46PM (#40361813)

Maybe if you tell it to pull the answer from a limerick instead of generally asking?

Edit: Ok no, I tried giving it a whole bunch of hints, and it was just making stuff up that was completely unrelated. Even directly pointing it at the original dataset didn’t help.

[ Parent | on HN ]

Re:GPT-4o's Memory Breakthrough (Needle in a Needlestack) by causal on Tuesday May 14, 2024 @11:27PM (#40362784)

Yeah I also tried to get it to complete some limericks from the dataset. Curiously it believed it had heard of the limerick but would then recite a hallucination.

So the good news is that the NIAN score might be real, bad news is you can't rely on it to know what it knows.

[ Parent | on HN ]

Re:GPT-4o's Memory Breakthrough (Needle in a Needlestack) by seanhunter on Wednesday May 15, 2024 @02:21PM (#40370524)

If you ask it to complete a limerick and it finishes it differently from the original, but it still works as a limerick is that really a hallucination?

[ Parent | on HN ]

Re:GPT-4o's Memory Breakthrough (Needle in a Needlestack) by EGreg on Tuesday May 14, 2024 @09:49PM (#40362246)

Come on guys, it’s already far beyond superhuman if it’s able to do that and so quickly. So if it’s not able to do that, what’s the big deal? If you’re asking for AG.I., then it seems that the model performs beyond it in these areas.

[ Parent | on HN ]

Re:GPT-4o's Memory Breakthrough (Needle in a Needlestack) by Aeolun on Thursday May 16, 2024 @01:15AM (#40375412)

We were mainly trying to determine if there was a reasonable chance that the model was trained on a certain dataset, nothing else.

[ Parent | on HN ]