News for hackers. Stuff that matters.

A comment on GPT-3 Is “Mindblowing” If You Don’t Question It Too Closely. Parent comment

Re:GPT-3 Is “Mindblowing” If You Don’t Question It Too Closely by SpicyLemonZest on Sunday August 16, 2020 @02:53PM (#24180180)
You'd have to study the GPT architecture to answer that question. For example, there's a comment upthread describing how its arithmetic capacity may be artificially low due to the way it parses math problems. I've seen another analysis, I forget where, explaining that it's fundamentally incapable of certain kinds of complex reasoning because the sequence of transformations producing the output is static and non-rewindable.

While the article's extreme skepticism is unwarranted, most experts do seem to agree that GPT can't scale to general human-level intelligence.

[ Parent | on HN ]
Re:GPT-3 Is “Mindblowing” If You Don’t Question It Too Closely by wombatmobile on Sunday August 16, 2020 @03:16PM (#24180384)
Thank you for a considered response, SpicyLemonZest. When I posted that question I half expected to be howled down as a troll, a naysayer, or off-topic. I'm glad you did none of that and answered by sharing your thoughts. As humans in the community of HN, we are richer for it.

> most experts do seem to agree that GPT can't scale to general human-level intelligence.

What is it about human-level intelligence that's unreachable by machines, even machines as advanced as V42 of GPT, would you consider? Can you put your finger on it, or make a little start?

[ Parent | on HN ]
Re:GPT-3 Is “Mindblowing” If You Don’t Question It Too Closely by SpicyLemonZest on Sunday August 16, 2020 @03:36PM (#24180550)
I doubt there's any property of human-level intelligence that's inherently unreachable by machines. The question is whether GPT-42 would actually be tremendously advanced, or whether we're close to the limit of what the architecture can produce.

The consensus among experts I've read is that known architectural limitations make the GPT-n series unlikely to scale beyond "sounds very human if you don't question it too closely". Algebra, to pick a concrete example, is something that there's reason to believe the GPT-series can't learn effectively - the output generation mechanism is fundamentally incapable of looping or recursion, which severely limits how well it can break things down into subproblems.

[ Parent | on HN ]