News for hackers. Stuff that matters.

A comment on Kolmogorov-Arnold Networks. Parent comment

Re:Kolmogorov-Arnold Networks by mp187 on Wednesday May 01, 2024 @04:57AM (#40220987)
Why was this your first thought? Is a limiting factor to transformers the MLP layer? I thought the bottleneck was in the renormalization part.
[ Parent | on HN ]
Re:Kolmogorov-Arnold Networks by brrrrrm on Wednesday May 01, 2024 @09:10AM (#40222676)
At small input size, yes the MLP dominates compute. At large input attention matters more
[ Parent | on HN ]