The Deep Research Problem

Re:The Deep Research Problem by simonw on Tuesday February 25, 2025 @06:32PM (#43178882)

When ChatGPT came out, one of the things we learned is that human society generally assumes a strong correlation between intelligence and the ability to string together grammatically correct sentences. As a result, many people assumed that even GPT-3.5 was wildly more "intelligent" than it actually was.

I think Deep Research (and tools like it) offer an even stronger illustration of that same effect. Anything that can produce a well-formatted multiple page report with headings and citations surely must be of PhD-level intelligence, right?

(Clearly not.)

[ Parent | on HN ]

Re:The Deep Research Problem by SubiculumCode on Tuesday February 25, 2025 @11:30PM (#43180612)

In some ways, it's a good tool to teach yourself to sus out the real clues to reliability, not format and authoritative tone.

[ Parent | on HN ]

Re:The Deep Research Problem by j_maffe on Wednesday February 26, 2025 @03:56AM (#43182025)

But that's the thing. The only way to truly find out if it's reliable (>90%) is to check the data yourself.

[ Parent | on HN ]

Re:The Deep Research Problem by SubiculumCode on Wednesday February 26, 2025 @11:09AM (#43184995)

This is why metrics and leaderboards like these are so important (but under reported on): https://github.com/vectara/hallucination-leaderboard https://www.kaggle.com/facts-leaderboard

Google Gemni models seem to lead...hopefully the metrics aren't being gamed.

[ Parent | on HN ]

Re:The Deep Research Problem by immibis on Tuesday February 25, 2025 @11:32PM (#43180620)

That's been every LLM since GPT-2.

[ Parent | on HN ]

Re:The Deep Research Problem by isaiahwp on Wednesday February 26, 2025 @04:51AM (#43182296)

To be fair, OpenAI's the one marketing it as such.

[ Parent | on HN ]

Re:The Deep Research Problem by jcgrillo on Wednesday February 26, 2025 @10:36AM (#43184628)

computer use big big words ergo computer rull rull smrt

[ Parent | on HN ]