News for hackers. Stuff that matters.

A comment on Neural Networks and the Chomsky Hierarchy [DeepMind, 2022]. Parent comment

Re:Neural Networks and the Chomsky Hierarchy [DeepMind, 2022] by ben_w on Monday January 23, 2023 @09:31AM (#34489165)
I don't know what point you're making, GPT-3 almost certainly has training examples for greater depth.

Pattern recognition can fail. Mine does, GPT's does.

I count further than I subitize, but from experience my mind may wander and I may lose track after 700 of a thing even if there's literally nothing else to focus on.

But I still didn't notice an error at depth 3.

[ Parent | on HN ]
Re:Neural Networks and the Chomsky Hierarchy [DeepMind, 2022] by cma on Monday January 23, 2023 @09:55AM (#34489459)
It seems transformers don't generalize in this class and neural turing machines and neural stack machines do.

You get the general rule (in linguistics known as competency) but can't flawlessly do it (performance). Transformers can't seem to get competence here.

https://en.m.wikipedia.org/wiki/Linguistic_competence

[ Parent | on HN ]
Re:Neural Networks and the Chomsky Hierarchy [DeepMind, 2022] by ben_w on Monday January 23, 2023 @05:34PM (#34496092)
Ah, gotcha. Thanks for the clear explanation and the link! :)
[ Parent | on HN ]