But the main issue with GPUs (or FPGAs / ASICs) is now you need to send 9GBps to some other chip AND back again.

Which means 9GBps downstream (to be processed by the GPU) + 9GBps upstream (GPU is done with the data), or a total bandwidth of 18GBps aggregate to the GPU / FPGA / ASIC / whatever coprocessor you're using.

So that's what? Another 32x lanes of PCIe 3.0? Maybe a 16x PCIE 4.0 GPU can handle that kind of I/O... but you can see that moving all this data around is non-trivial, even if we assume the math is instantaneous.

---------

Practically speaking, it seems like any CPU with enough PCIe bandwidth to handle this traffic is a CPU beefy enough to seemingly run the math.

PCIE 3.0 is 1GB/s per lane in each direction. A 3.0 8x link would do a good job of saturating the drives. And basically any CPU could run 8x to the storage controllers and 8x to a GPU. Get any Ryzen chip and you can run 4 lanes directly to a network card too.

If only the ASIC on the HDD could run these computations and correct bit errors right during data transfers!

The HDD ASIC certainly is doing those computations.

The issue is that Backblaze has a 2nd layer of error correction codes. This 2nd layer of error correction codes needs to be calculated somewhere. If enough errors come from a drive, the administrators take down the box and replace the hard-drives and resilver the data.

Backblaze physically distributes the data over 20 separate computers in 20 separate racks. Some computer needs to run the math to "Combine" the data (error correction + checksums and all) back into the original data on every single read. So singular hard drive can do this math because the data has been reliably dispersed to so many different computers.