News for hackers. Stuff that matters.
Re:Every Model Learned by Gradient Descent Is Approximately ... by smallnamespace on Sunday December 06, 2020 @08:51PM (#25328609)
Most definitely, what I meant though is how do we know mental attention is not essentially a big softmax (over some context, e.g. short or long term memory)?

The eye physically focusing on one thing at time seems like a special case (and this fact isn't even true of all animals, e.g. many prey species), and not a part of the brain's attention mechanism.

Re:Every Model Learned by Gradient Descent Is Approximately ... by Straw on Monday December 07, 2020 @06:55AM (#25331307)
Oh, I don't know, it very well might! However, the brain's physical structure doesn't look conducive to 4096x4096 matmuls!