News for hackers. Stuff that matters.
Re:Deep Research by YmiYugy on Sunday February 02, 2025 @07:33PM (#42913462)
If I understood the graphs correctly, it only achieves 20% pass rate on their internal tests. So I have to wait 30min and pay a lot of money just to sift through walls of most likely incorrect text? Unless the possibility of hallucinations is negligible, this is just way too much content to review at once. The process probably needs to be a lot more iterative.
Re:Deep Research by brokensegue on Sunday February 02, 2025 @07:46PM (#42913574)
26.6% on humanity's last exam is actually impressive.

pass rate really only matters in context of the difficulty of the tasks

Re:Deep Research by tmnvdb on Sunday February 02, 2025 @07:36PM (#42913483)
Only if you are asking questions at the level of a cutting edge benchmark
Re:Deep Research by rvnx on Sunday February 02, 2025 @07:50PM (#42913614)
This is one of the actual questions:

> In Greek mythology, who was Jason's maternal great-grandfather?

https://www.google.com/search?q=In+Greek+mythology%2C+who+wa...

Re:Deep Research by tmnvdb on Sunday February 02, 2025 @08:07PM (#42913751)
This is a hard question for language models since it targets one of their known weaknesses.
Re:Deep Research by layer8 on Sunday February 02, 2025 @08:15PM (#42913804)
Users don’t care about how hard something is for LLMs if they receive incorrect output.
Re:Deep Research by andyg_blog on Sunday February 02, 2025 @08:21PM (#42913858)
Greek mythology? But seriously please elaborate for my less educated self.
Re:Deep Research by _bin_ on Sunday February 02, 2025 @11:21PM (#42914950)
it tests syllogistic reasoning: Jason's mother was Tyro, whose father was Poesidon, whose father was Kronos. it also tests whether it "eagerly" rather than comprehensively considers something: a maternal great-grandfather could be the father of either one's maternal grandmother or maternal grandfather. so the answer could also be king Aeolus of the Etruscans.

ideally a model would be able to answer this accurately and completely.

Re:Deep Research by nimithryn on Sunday February 02, 2025 @11:46PM (#42915082)
I think there are more possible answers? Jason's mother differs depending on the author...

For example, Jason's mother was Philonis, daughter of Mestra, daughter of Daedalion, son of Hesporos. So Jason's maternal great-grandfather was Hesporos.

Re:Deep Research by tmnvdb on Sunday February 02, 2025 @08:51PM (#42914055)
LLMs often don't do well on tasks that require composition into smaller subtasks. In this case there is a chain of relations that depend on the previous result.
Re:Deep Research by 11101010001100 on Sunday February 02, 2025 @08:23PM (#42913875)
It's categorically more than a weakness.
Re:Deep Research by elicksaur on Sunday February 02, 2025 @08:08PM (#42913754)
In Greek mythology, Jason's maternal great-grandfather was Einstein.
Re:Deep Research by pama on Sunday February 02, 2025 @09:10PM (#42914173)
No it is not an actual question on this exam. From the paper: “To ensure question quality and integrity, we enforce strict submission criteria. Questions should be precise, unambiguous, solvable, and non-searchable, ensuring models cannot rely on memorization or simple retrieval methods. All submissions must be original work or non-trivial syntheses of published information, though contributions from unpublished research are acceptable. Questions typically require graduate-level expertise or test knowledge of highly specific topics (e.g., precise historical details, trivia, local customs) and have specific, unambiguous answers…”. (Emphasis mine)
Re:Deep Research by Neynt on Sunday February 02, 2025 @10:03PM (#42914454)
It's example #7 on https://lastexam.ai/
Re:Deep Research by pama on Monday February 03, 2025 @10:13AM (#42918951)
This is an example of the submitted questions. Because it is possible to search it on the web, it is not an example of the accepted questions.
Re:Deep Research by freehorse on Monday February 03, 2025 @03:47AM (#42916230)
I am selling a bridge, it is a great bargain.
Re:Deep Research by johnfn on Sunday February 02, 2025 @11:35PM (#42915030)
Did you intentionally flip through all the questions to find the one that seemed the easiest? If so, why? That's question #7, and all other 7 questions in the sample set seem ridiculously difficult to me.
Re:Deep Research by roenxi on Sunday February 02, 2025 @08:00PM (#42913698)
Maybe. Not enough data to say. Say it does a days worth of work in a query. It is sensible to use if it takes less than a day to review ~5 days worth of work. I don't know if we're near that threshold yet but conceptually this would work well for actual research where the amount of preparation is large compared to the amount of output written.

And eyeballing the benchmarks, it'll probably reach a >50% rate per query by the end of the year. Seems to double every model or two.

Re:Deep Research by spyckie2 on Sunday February 02, 2025 @07:59PM (#42913685)
I mean you want it to grill your steak and eat it for you too?

I mean I too can complain that my iPhone doesn’t automatically screen out spammers and send my mom flowers on Mother’s Day.

Re:Deep Research by scarab92 on Sunday February 02, 2025 @08:13PM (#42913788)
Why doesn't the iPhone screen spammers yet? Pixel has had this feature for a decade.
Re:Deep Research by senordevnyc on Monday February 03, 2025 @03:07AM (#42916014)
Pixel hasn’t even been around for a decade.
Re:Deep Research by scarab92 on Monday February 03, 2025 @05:31AM (#42916851)
The Pixel branding is 12 years old, and IIRC this feature also existed in Nexus before that.
Re:Deep Research by senordevnyc on Monday February 03, 2025 @11:35AM (#42919989)
Haha, are you referring to the Chromebook Pixel? How is that relevant to stopping spam calls?

Pixel phone launched in 2016.

Re:Deep Research by itkovian_ on Sunday February 02, 2025 @08:22PM (#42913867)
Here's an example of the type of question it is acheiving 20% on;

The set of natural transformations between two functors F,G ⁣:C→DF,G:C→D can be expressed as the end Nat(F,G)≅∫AHomD(F(A),G(A)). Nat(F,G)≅∫A HomD (F(A),G(A)).

Define set of natural cotransformations from FF to GG to be the coend CoNat(F,G)≅∫AHomD(F(A),G(A)). CoNat(F,G)≅∫AHomD (F(A),G(A)).

Let: - F=B∙(Σ4)∗/F=B∙ (Σ4 )∗/ be the under ∞∞-category of the nerve of the delooping of the symmetric group Σ4Σ4 on 4 letters under the unique 00-simplex ∗∗ of B∙Σ4B∙ Σ4 . - G=B∙(Σ7)∗/G=B∙ (Σ7 )∗/ be the under ∞∞-category nerve of the delooping of the symmetric group Σ7Σ7 on 7 letters under the unique 00-simplex ∗∗ of B∙Σ7B∙ Σ7 .

How many natural cotransformations are there between FF and GG?

Re:Deep Research by Davidzheng on Sunday February 02, 2025 @10:17PM (#42914528)
btw isn't this question at least really badly worded (and maybe incorrect?) the definitions they give for F and G are categories not functors... (and both categories are in fact one object with contractible space of morphisms...)
Re:Deep Research by perching_aix on Sunday February 02, 2025 @11:27PM (#42914988)
It's very interesting to think about what kind of "mental model" might it have, if it's capable of "understanding" all this (to me) gibberish, but is then unable to actually work the problem.
Re:Deep Research by slaterbug on Monday February 03, 2025 @01:17AM (#42915478)
As someone who doesn't understand anything beyond the word 'set' in that question, can anyone give an indication of how hard of a problem that actually is (within that domain)?

Also I'm curious as to what percentage of the questions in this benchmark are of this type / difficulty, vs the seemingly much easier example of "In Greek mythology, who was Jason's maternal great-grandfather?".

I'd imagine the latter is much easier for an LLM, and almost trivial for any LLM with access to external sources (such as deep research).

Re:Deep Research by baal80spam on Monday February 03, 2025 @02:51AM (#42915923)
That's easy Dave: 42.
Re:Deep Research by YmiYugy on Monday February 03, 2025 @01:38PM (#42921302)
Do we actually know whether it got this specific example right? It got 20% on HLE, but I think a few questions are quite a bit easier.
Re:Deep Research by dyauspitr on Sunday February 02, 2025 @10:28PM (#42914603)
On questions even specialists in that field can’t answer correctly.
Re:Deep Research by throwaway123lol on Monday February 03, 2025 @10:12AM (#42918940)
Yeah it can be more iterative. Just use individual queries and build on it yourself. This is all this is doing. It's a trick, and OpenAI is a PR hype company at this stage.