News for hackers. Stuff that matters.

A comment on Cosma Shalizi on Attention and Transformers. Parent comment

Re:Cosma Shalizi on Attention and Transformers by x1000 on Sunday December 24, 2023 @08:17PM (#38758681)
There’s a video[1] of Karpathy recounting an email correspondence he had with with Bahdanau. The email explains that the word “Attention” comes from Bengio who, in one of his final reviews of the paper, determined it to be preferable to Bahdanau’s original idea of calling it “RNNSearch”.

[1] https://youtu.be/XfpMkf4rD6E?t=18m23s

[ Parent | on HN ]
Re:Cosma Shalizi on Attention and Transformers by behnamoh on Sunday December 24, 2023 @08:49PM (#38758842)
"RNNSearch is all you need" probably wouldn't catch on and we'd still be ChatGPT-less.
[ Parent | on HN ]
Re:Cosma Shalizi on Attention and Transformers by rounakdatta on Monday December 25, 2023 @12:38AM (#38760129)
Nerds pay attention nevertheless.
[ Parent | on HN ]
Re:Cosma Shalizi on Attention and Transformers by TeMPOraL on Monday December 25, 2023 @02:56AM (#38760702)
Worked with PageRank and "map reduce", tho.
[ Parent | on HN ]