kraken12

The 4090 has half the memory bandwidth, so it could not get a 5X gain, it would actually run slower on a memory bound LLM like this.

Yeah, it is an architectural simulation study, this is what is usually done right at the beginning before resources are allocated to go deep on idea. So in that sense it is imaginary; but this is how new ideas get incubated.

Yeah, a preliminary architectural study to sanity check if an idea could potentially pay off.

Maybe they could do something like AMD's GPU memory stacking, that is good for scaling, and of course they are using many chips not one chip..

Seems like HN comments have determined that the cost number is not fudged..

Seems to me the 18 tokens per second from [1] is the throughput and includes the batch size, so I don't think they misread the Deepspeed inference paper. So the chiplet ASIC supercomputer paper would seem to show a decent performance/TCO benefit.

Of course, it's a first architectural study to illustrate the promise of the idea, lots more details to work out in a physical implementation and the final realized benefit is likely to be lower. But even a 3X is huge in this space.

Yep, it's a research paper in comp arch, the initial proof-of-concept study before you go and spend real money on it.

They are using many chips and taking advantage of the way data flows in LLMs to make it work; so it would be cost-effective unlike Cerebras