I’m not seeing anything insightful though. They are taking a fairly standard architecture, claiming it’s something different, and then claiming some connection to biological neurons.

As I read it, this is an example of people publishing a minor tweak in the algorithm and then claiming it’s a new biologically inspired abstraction. I guess I just want more papers about transformers.