Neural Networks and the Chomsky Hierarchy [DeepMind, 2022]

Re:Neural Networks and the Chomsky Hierarchy [DeepMind, 2022] by ben_w on Monday January 23, 2023 @02:44AM (#34486446)

I've seen humans mess up matching braces. I've even been that human already this year, only finding out because the compiler told me.

[ Parent | on HN ]

Re:Neural Networks and the Chomsky Hierarchy [DeepMind, 2022] by randorand0 on Monday January 23, 2023 @05:34AM (#34487303)

this may well be the only real skill of programmers who learned C style syntaxes, andor LISPs

on the ohter hand, thinking with the parenthesis takes language up to the level of functional-thinking. computable stuff is all about substitution of all things inside a parenthesis for a single thing. and this is the essence of all classic computation

[ Parent | on HN ]

Re:Neural Networks and the Chomsky Hierarchy [DeepMind, 2022] by cma on Monday January 23, 2023 @08:45AM (#34488633)

But we still generalize it to bigger and bigger without seeing matching brace problems of all sizes we can handle.

In the paper I don't think they are looking for perfect, but they show which models can't seem to learn significantly beyond the size of examples they explicitly saw.

[ Parent | on HN ]

Re:Neural Networks and the Chomsky Hierarchy [DeepMind, 2022] by ben_w on Monday January 23, 2023 @09:09AM (#34488860)

The {specific (number I (messed up was a depth of 3)}.

I missed it because it was spaced over a few lines and I was tired.

[ Parent | on HN ]

Re:Neural Networks and the Chomsky Hierarchy [DeepMind, 2022] by cma on Monday January 23, 2023 @09:20AM (#34489004)

But not because you only trained up to examples of two.

[ Parent | on HN ]

Re:Neural Networks and the Chomsky Hierarchy [DeepMind, 2022] by ben_w on Monday January 23, 2023 @09:31AM (#34489165)

I don't know what point you're making, GPT-3 almost certainly has training examples for greater depth.

Pattern recognition can fail. Mine does, GPT's does.

I count further than I subitize, but from experience my mind may wander and I may lose track after 700 of a thing even if there's literally nothing else to focus on.

But I still didn't notice an error at depth 3.

[ Parent | on HN ]

Re:Neural Networks and the Chomsky Hierarchy [DeepMind, 2022] by cma on Monday January 23, 2023 @09:55AM (#34489459)

It seems transformers don't generalize in this class and neural turing machines and neural stack machines do.

You get the general rule (in linguistics known as competency) but can't flawlessly do it (performance). Transformers can't seem to get competence here.

https://en.m.wikipedia.org/wiki/Linguistic_competence

[ Parent | on HN ]

Re:Neural Networks and the Chomsky Hierarchy [DeepMind, 2022] by ben_w on Monday January 23, 2023 @05:34PM (#34496092)

Ah, gotcha. Thanks for the clear explanation and the link! :)

[ Parent | on HN ]