There's a lot of progress on optical computing recently, with some of its proponents suggesting they could get up to 100x speedup on some things like LLMs relative to current chips. If they ever manage to deliver then we'll see a lot of 2x or more improvements.

Even if we achieve 2x every 3-4 years, isn't it still massive in absolute ?

Yes, exponential growth gets quite large quite fast. Think about how a $300 graphics card today competes against one from 15 years ago. They're orders of magnitude faster, more memory, etc.

Headline marketing "AI TOPS" is also based on whatever smaller format a new generation supports. E.g. doubling headline marketing performance from Ada to Blackwell by going from FP8 to FP4.

Don't forget "structured sparsity" where two out of every four weights must be zero. Another trick that Nvidia still happily assume for the sake of a graph or an headline figure.

Everything has growth limits. For me Moore's law is number of transistor per dollar, not number of transistor per unit area. And even if we don't have any technical progress that number will decrease as the fab would recoup the R and D costs and the competition will increase after technical saturation.

We would likely get 2x transistor per dollar every two year for quite a long time.

It's always been a function of price, I've been shouting this for years: https://arstechnica.com/gadgets/2008/09/moore/

> The complexity for minimum component costs has increased at a rate of roughly a factor of two per year.

It's incredible how deeply ingrained this misinterpretation has been, even among the very technical, without ever examining its original source or phrasing.