Because in HFT, every fraction of a second can mean the difference between 10s or even 100s of millions of dollars in profit/loss.
What GP means is that the portion of the overall latency which you can control must respond as quickly as possible, and you do count nanoseconds. Obviously HFT has not invented superluminal communication techniques.
They haven't, but they've put in tremendous resources into going above the speed of light in optical fibre, including the use of point-to-point microwave links and, when it's impossible to place towers for microwave links (for example, across the Atlantic Ocean), the use of HF/shortwave (as in 3-30MHz stuff) radio.
Light can only go a few km before it needs to be boosted
The speed of light in a fiber is only about 70% of speed of light in vacuum.
Modern single mode fiber has very low attenuation (less than 0.01dB/km). But more importantly you don't have to use opto-electrical repeaters to boost the signal. What you do is using purely passive dispersion control by mixing different kinds of fiber and compensate for the rest dispersion using chirped fiber bragg gratings. To boost the signal you splice in rare earth doped fibers (Erbirum, Yttrium) which acts as optically pumped optical amplifier.
/me trying to get a handle on optical development.
They're used in long haul communication for some 15 years or so. They're especially useful in submarine cables, for the sole reason, that new optical transmission schemes(modulation, wavelength multiplexing and so on) can be implemented by just replacing the equipment on either end, without the need to replace components down there on the seabed.
Two issues with this:
1) You cannot compare an FPGA architecture to a general purpose CPU. FPGAs are massively parallel by design and have near zero overhead, so 100 MHz in an FPGA != a 100 MHz CPU.
2) FPGAs do not lag that far behind and are moving to 16 nm and beyond for higher end products. Besides, due to 1), there really is no real need to move to lower nodes yet.
But more importantly, FPGAs are parallel in a way conventional processors aren't.