It’s a bet. Probably a good one to make. The upside of being the ones who have an AI chip (not a graphics chip larping as an AI chip) is huge. It will run faster and more cheaply. You get to step all over OpenAI, or get a multi billion dollar deal to supply Microsoft data centres. Or these ship on every new laptop etc. You get to be the next unicorn ($1tn company). So that is a decent bet for investors assuming the team can deliver. Yes the danger is there is a new architecture that runs on a CPU that for practical purposes whoop’s Attention’s ass. In which case investors can throw some money at ASICifying that.

Yep, transformers showed up in 2017, nearly 7 years ago, and they still wear the crown. Maybe some new architecture will come to dominate eventually, but I would love a low cost PCIe board that could run 80B transformer models today.

If the fairy tale numbers are correct, they could price it at a million dollars and it would still be cheap.

This was my first thought too. Even if transformers turn out to be the holy grail for LLMs, people are still interested in diffusion models for image generation.

I think we’re about to see a lot of interesting specialized silicon for neural nets in the coming years, but locking yourself into a specific kind of model seems a little too specialized right now.

Diffusion models could actually be implemented with transformers, hypothetically. Their training and inference is what makes diffusion models unique, not the model architecture.

Well, GPT-4 runs on a transformer architecture, and even if for unknown reasons GPT-4 is the upper limit of what you can achieve with transformer models, having hardware specialized to run the architecture extremely fast would always be very useful for many tasks (the tasks GPT-4 can already handle at least).