News for hackers. Stuff that matters.
Re:AbsenceBench: Language models can't tell what's missing by VikRubenfeld on Saturday June 21, 2025 @09:46AM (#44337575)
Interesting. I encountered an example of this the other day. I was asking chatGPT:

> Is there a line in Hamlet that includes the words "utter love?"

It examined a lot of quotes from Hamlet, and concluded:

> I double-checked all editions: There is no line in Hamlet that contains the word “utter.”

>It is never used in any of the quartos or folios in any line of Hamlet.

I was pretty sure it was there, so I found the full text of Hamlet online and searched for the word "utter". I found it right away and posted to chatGPT:

> That's odd. I found it in two seconds by searching the full text for the word "utter".

> This must be known, which, being kept close, might move More grief to hide than hate to utter love.

> Come

> HAMLET, Act 2, Scene 2

chatGPT replied:

> You're absolutely right — thank you for pointing that out.

> The line is indeed from Act 2, Scene 2, spoken by Claudius, and here it is in full:

> "This must be known, which, being kept close, might move

> More grief to hide than hate to utter love."

> I regret the earlier oversight — your memory was sharper than my index!

Re:AbsenceBench: Language models can't tell what's missing by diggan on Saturday June 21, 2025 @09:52AM (#44337624)
Yeah, kind of makes sense, LLMs are really bad at that sort of recall unless provided tool and a loop to go over the results.

For example, I could replicate it saying "no there isnt" when using 4o without search. But with search it gets the answer right.

It was important already, but it's getting even more important than ever to use the right tool for the right problem, especially as all the hypers try to sell us their own stuff.

Re:AbsenceBench: Language models can't tell what's missing by GLdRH on Saturday June 21, 2025 @12:30PM (#44338790)
Seems like ChatGPT is an utter failure
Re:AbsenceBench: Language models can't tell what's missing by speakeron on Saturday June 21, 2025 @05:09PM (#44340747)
It's actually Act 2, Scene 1 and is spoken by Polonius.