If it were just a matter of doing that, we would be fine with fully-connected MLP. And maybe that would work with orders of magnitude more data and compute than we currently throw at these models, we are already pushing the cutting edge of those things to get useful results out of the specialized architecture.
Choosing the right NN architecture is like feature engineering: the exact details don't matter that much, but getting the right overall structure can be the difference between learning a working model and failing to learn a working model, from the same source data with the same information content. Clearly our choice of inductive bias matters, and the transformer architecture is clearly an improvement over other designs.
Surely you wouldn't argue that a CNN is "just" aligning computation to hardware, right? Transformers are clearly showing themselves as a reliably effective model architecture for text in the same way that CNNs are reliably effective for images.
It was an innovation, in the 80s, to map image structure to weight structure that underpins CNNs. That isnt what made CNNs trainable though.. that was alexnet, and just go read the paper... its pretty upfront about how the NN architecture is designed to fit the GPU... that's the point of it
Isn't that essentially what mjburgess said in the parent post?
> LLMs are a hardware innovation: they make it possible for GPUs to compute this at scale across TBs of data... The algorithm isnt doing anything other than aligning computation to hardware