News for hackers. Stuff that matters.
Re:'Attention is all you need' coauthor says he's 'absolutel... by skissane on Saturday October 25, 2025 @12:18AM (#45701330)
> Add a dedicated "parking spot" like GPT-OSS does and eat the gradient flow tax on that

Not familiar with this topic, but intrigued-anywhere I can read more about it?

Re:'Attention is all you need' coauthor says he's 'absolutel... by ACCount37 on Saturday October 25, 2025 @08:48AM (#45703490)
Looked for it briefly, think the best I got is this older discussion:

https://news.ycombinator.com/item?id=44834918

Re:'Attention is all you need' coauthor says he's 'absolutel... by qcnguy on Saturday October 25, 2025 @01:40PM (#45705627)
OpenAI have talked about it. The neural architecture needs to let the model handle the case where there's nothing worth attending to, as softmax requires attention to be allocated to all tokens but sometimes there's nothing worth it.