How to Scale Your Model: A Systems View of LLMs on TPUs

Re:How to Scale Your Model: A Systems View of LLMs on TPUs by awongh on Tuesday February 04, 2025 @04:07PM (#42938709)

Here in the thread he says: https://x.com/jacobaustin132/status/1886844724339675340 : `5 years ago, there were many ML architectures, but today, there is (mostly) only one [transformers].`

To what degree is this actually true, and what else is on the horizon that might become as popular as transformers?

[ Parent | on HN ]

Re:How to Scale Your Model: A Systems View of LLMs on TPUs by swyx on Wednesday February 05, 2025 @01:45AM (#42944784)

it's quite true. the convergence of all archs to transformers is well documented by karpathy. SSMs were once touted as transformer killers, but increasingly look like just optional supplements.

[ Parent | on HN ]