News for hackers. Stuff that matters.
Re:Deep Research by rvnx on Sunday February 02, 2025 @07:50PM (#42913614)
This is one of the actual questions:

> In Greek mythology, who was Jason's maternal great-grandfather?

https://www.google.com/search?q=In+Greek+mythology%2C+who+wa...

Re:Deep Research by tmnvdb on Sunday February 02, 2025 @08:07PM (#42913751)
This is a hard question for language models since it targets one of their known weaknesses.
Re:Deep Research by layer8 on Sunday February 02, 2025 @08:15PM (#42913804)
Users don’t care about how hard something is for LLMs if they receive incorrect output.
Re:Deep Research by andyg_blog on Sunday February 02, 2025 @08:21PM (#42913858)
Greek mythology? But seriously please elaborate for my less educated self.
Re:Deep Research by _bin_ on Sunday February 02, 2025 @11:21PM (#42914950)
it tests syllogistic reasoning: Jason's mother was Tyro, whose father was Poesidon, whose father was Kronos. it also tests whether it "eagerly" rather than comprehensively considers something: a maternal great-grandfather could be the father of either one's maternal grandmother or maternal grandfather. so the answer could also be king Aeolus of the Etruscans.

ideally a model would be able to answer this accurately and completely.

Re:Deep Research by nimithryn on Sunday February 02, 2025 @11:46PM (#42915082)
I think there are more possible answers? Jason's mother differs depending on the author...

For example, Jason's mother was Philonis, daughter of Mestra, daughter of Daedalion, son of Hesporos. So Jason's maternal great-grandfather was Hesporos.

Re:Deep Research by tmnvdb on Sunday February 02, 2025 @08:51PM (#42914055)
LLMs often don't do well on tasks that require composition into smaller subtasks. In this case there is a chain of relations that depend on the previous result.
Re:Deep Research by 11101010001100 on Sunday February 02, 2025 @08:23PM (#42913875)
It's categorically more than a weakness.
Re:Deep Research by elicksaur on Sunday February 02, 2025 @08:08PM (#42913754)
In Greek mythology, Jason's maternal great-grandfather was Einstein.
Re:Deep Research by pama on Sunday February 02, 2025 @09:10PM (#42914173)
No it is not an actual question on this exam. From the paper: “To ensure question quality and integrity, we enforce strict submission criteria. Questions should be precise, unambiguous, solvable, and non-searchable, ensuring models cannot rely on memorization or simple retrieval methods. All submissions must be original work or non-trivial syntheses of published information, though contributions from unpublished research are acceptable. Questions typically require graduate-level expertise or test knowledge of highly specific topics (e.g., precise historical details, trivia, local customs) and have specific, unambiguous answers…”. (Emphasis mine)
Re:Deep Research by Neynt on Sunday February 02, 2025 @10:03PM (#42914454)
It's example #7 on https://lastexam.ai/
Re:Deep Research by pama on Monday February 03, 2025 @10:13AM (#42918951)
This is an example of the submitted questions. Because it is possible to search it on the web, it is not an example of the accepted questions.
Re:Deep Research by freehorse on Monday February 03, 2025 @03:47AM (#42916230)
I am selling a bridge, it is a great bargain.
Re:Deep Research by johnfn on Sunday February 02, 2025 @11:35PM (#42915030)
Did you intentionally flip through all the questions to find the one that seemed the easiest? If so, why? That's question #7, and all other 7 questions in the sample set seem ridiculously difficult to me.