News for hackers. Stuff that matters.

A comment on Transformer learning explained: Coinductive guide to inductive transformer heads. Parent comment

Re:Transformer learning explained: Coinductive guide to indu... by adamnemecek on Tuesday February 28, 2023 @11:30AM (#34971222)
Yes, that's the reason why I'm talking about it. It should be faster since it works locally (within the single layers) as opposed to across the whole graph.
[ Parent | on HN ]
Re:Transformer learning explained: Coinductive guide to indu... by rnosov on Tuesday February 28, 2023 @11:36AM (#34971317)
Could you quantify it (Twice, three times, etc)? Another question, does it apply to only attention heads learning or the whole shebang?
[ Parent | on HN ]
Re:Transformer learning explained: Coinductive guide to indu... by adamnemecek on Tuesday February 28, 2023 @11:39AM (#34971366)
It's going to be more than that, like significantly. Hopf coherence theoretically makes away with backprop altogether.

I'm aware of Hinton's forward-forward, but I need more time to provide a half-decent comparison.

[ Parent | on HN ]