News for hackers. Stuff that matters.
Re:Is the reversal curse in LLMs real? by foobarqux on Friday November 17, 2023 @11:41AM (#38305979)
Humans have reasoning capabilities that are separate from memory. If you want to show that LLMs have reasoning capability you try to create carefully constructed adversarial experiments that try to distinguish between that hypothesis and others, you do not just show that there are cases that favor your hypothesis.

And although it's a complete waste of time because the answer is obvious when you do do these types of adversarial experiments, like adversarial blockworld, you find that LLMs are not internalizing abstract principles inherent to reasoning and are instead doing something like string-pattern-patching.

Re:Is the reversal curse in LLMs real? by foobarqux on Friday November 17, 2023 @08:43PM (#38313818)
edit: I meant to say "consistent with your hypothesis" not "favor your hypothesis"