The other thing came from my old, favorite, supercomputing vendor: SGI. They used to blow my mind repeatedly with how they did hardware. On the Altix NUMA systems, they had an option called RASC nodes that basically added 2 FPGA's plus QDR RAM to the system. Like everything else, the FPGA's attached to the shared memory bus to operate on whole memory directly with microseconds latency and several GB/s bandwidth. Not sure if modern FPGA's are sitting on the memory bus inexpensively but RASC's performance/integration showed it's a great idea.
So, those were my two fun points on the subject. The next will probably be if I land a Pico Computing desktop w/ some high-end FPGA's or if Archipelago gets closer to mature FPGA architecture. Either have much potential. :)
The FPGA can give GPU and (especially!!) CPU a run for their money in the area of performance per watt.
The clock frequency gap is not relevant at all. It exists here as a side effect of some old wars between vendors. Some of recent CPU projects for exascale computing specifically reduce CPU frequency, trading it for predictability and reduced latency of operations. Both amount in less wasted cycles and in greater efficiency in the end.
Witness Xeon Phi, for example.
As for embedded devices... I don't know. My friends from embedded world used small FPGAs as a replacement for a lot of standalone logic chips (FPGA reduces set of 74xx chips to one). That's about it.
Much like CPUs you can't just compare Mhz X to Mhz Y, even moreso considering how parallel FPGAs can get.
There's also been some neat low-power parts lately(from lattice and others).
If we look at it from today, some things have changed, but the end result holds true - at least for general purpose computing.
The FPGA vendors (esp Xilinx and Altera) have released devices where the intergration between a hard CPU core and the FPGA is much tighter than using a standardised bus like PCI. This gives a much lower latency. In these devices the CPUs are normally ARM based.
The prevalence of GPUs as PCIe connected coporocessors and OpenCL has also made FPGA based computing connected via PCIe much easier.
One thing that has not improved, but rather worsened is the difference in clock frequency between CPU and FPGA. In Grays posting he talket about PPro running at 200 MHz. Today we have cores running at 2-3 GHz while FPGAs are hard to push beyond 200 MHz.
For applications that are highly parallel, takes a lot of cycles and not suitable for GPU processing, FPGAs might be useful as base for custom coprocessors even for x86, Power, SPARC based servers. But otherwise probably not.
For embedded systems FPGAs makes more sense, but cost of the device and power consumption might make it less so. The integrated SoC devices from Xilinx and Altera can do cool things like media processing, SDR competitively.
The Novena is an example of a modern system where the CPU has access to a FPGA over a low latency interface which makes it useful as a coprocessor. But the Novena is really not built to be cost efficient.