Re:Every Model Learned by Gradient Descent Is Approximately ...
by smallnamespace on Sunday December 06, 2020 @04:49PM
(#25326971)
How do we really know that brains use hard attention?
[ Parent
| on HN ]
Re:Every Model Learned by Gradient Descent Is Approximately ...
by Straw on Sunday December 06, 2020 @05:29PM
(#25327249)
Eyes only have high resolution in a tiny spot.
Re:Every Model Learned by Gradient Descent Is Approximately ...
by smallnamespace on Sunday December 06, 2020 @08:51PM
(#25328609)
Most definitely, what I meant though is how do we know mental attention is not essentially a big softmax (over some context, e.g. short or long term memory)?
The eye physically focusing on one thing at time seems like a special case (and this fact isn't even true of all animals, e.g. many prey species), and not a part of the brain's attention mechanism.
Re:Every Model Learned by Gradient Descent Is Approximately ...
by Straw on Monday December 07, 2020 @06:55AM
(#25331307)
Oh, I don't know, it very well might! However, the brain's physical structure doesn't look conducive to 4096x4096 matmuls!
Re:Every Model Learned by Gradient Descent Is Approximately ...
by l33tman on Monday December 07, 2020 @04:31AM
(#25330628)
Actually the retinal anisotropy is a feature distinct from attention. You can fixate on a point in a scene and then attend any other point in the scene. This is indeed how both animal and human attention experiments in the field is setup to control for the eyes movements.