If LLM can leverage on the new efficient attention mechanism based the FFT architecture discovered by Google then FPGA can be the new hot stuff [1]:

[1] The FFT Strikes Back: An Efficient Alternative to Self-Attention (168 comments):

https://news.ycombinator.com/item?id=43182325

Its never going to be as efficient as ASIC and the LLM market is definitely big enough for ASICs to be viable.

This is past tense no?

There's been neural processing chips since before LLM craze [1].

[1]: https://en.wikipedia.org/wiki/Neural_processing_unit#History

I hear this a lot, but in my experience this isn't true at all.

A Versal AI Edge FPGA has a theoretical performance of 0.7TFLOPs just from the DSPs alone, while consuming less power than a Raspberry Pi 5 and this is ignoring the AI Engines, which are exactly the ASICs that you are talking about. They are more power efficient than GPUs, because they don't need to pretend to run multiple threads each with their own register files or hide memory latency by swapping warps. Their 2D NOC plus cascaded connections allow them to have a really high internal memory bandwidth in-between the tiles at low power.

What they are missing is processing in memory, specifically LPDDR-PIM for GEMV acceleration. The memory controllers simply can't deliver a memory bandwidth that is competitive with what Nvidia has and I'm talking about boards like Jetson Orin here.

Now compare that FPGA to Groq, SambaNova, or Cerebras. The ASICs are more efficient.

FPGAs are neither here nor there and will always be niche. If you need the same thing many times you make dedicated silicon. If you need many different things available all at once you use a normal CPU. Only if ASICs are too expensive and CPUs are too slow the FPGA can shine.

If FPGAs are competitive on perf/watt, why aren't they more widespread (other than crap software tooling)?

Honestly I've asked different hardware researchers this question and they all seem to give different answers.

They're competitive on perf/watt because they're designed to do one thing. But they're much more expensive than an ASIC, which, if also designed to do one thing would be better than the FPGA.