Re:Gary Marcus's Kafkaesque critique of GPT-3
by gwern on Monday September 07, 2020 @03:56PM
(#24402222)
Yes, pretty much. As Marcus admits in his writeup, even despite using doubly adversarially filtered GPT-3 errors, the final errors are due mostly to bad (no) prompting and then penalizing GPT-3 for not telepathically reading his mind in order to provide a serious response to a silly question.
I'm about halfway through the first 25 errors or so (it's very tedious so I've been putting it off), and as expected, GPT-3 is often correct zero-shot if you use even a trivial short prompt like 'This is a series of scenarios describing a human taking actions in the world, designed to test physical and common-sense reasoning.'
Re:Gary Marcus's Kafkaesque critique of GPT-3
by gwern on Monday September 07, 2020 @07:16PM
(#24403735)
The point about prompting also applies to the GPT-2 counterexamples Marcus published earlier this year, and conspicuously fails to discuss in this one, BTW.
I'm about halfway through the first 25 errors or so (it's very tedious so I've been putting it off), and as expected, GPT-3 is often correct zero-shot if you use even a trivial short prompt like 'This is a series of scenarios describing a human taking actions in the world, designed to test physical and common-sense reasoning.'