It's not like most people in the field are programming GPUs.

If a FPGA vendor made a FPGA solution (both the hardware and the software libraries to integrate with one or two machine learning frameworks) that did basic matrix/tensor calculations faster/cheaper than GPUs, then they'd be able to take a lot of market off nvidia. Users wouldn't have a need to program the FPGA directly if they can work at the level of matrix operations.

That means making a whole networked black-box, based on FPGA(s) and exposing an API for external use.

It's certainly possible to make. It's also a very expensive very specialized appliance. All of that to do some matrix manipulations.

People are buying large quantities of very expensive GPUs just to do some matrix manipulations, so why not FPGAs?

But the point is that you don't have that much FPGA-specific code - once someone does the matrix manipulations and the proper integration, everyone else can just run e.g. tensorflow code on it faster and/or cheaper without specific expertise; if a FPGA vendor can do this one-time investment in software tools, then they can compete for a slice of the large pie of hardware revenue that nvidia now has for itself.

But, assuming that "matrix manipulation" code has existed for FPGAs for a while, since verilog and VHDL are quite old, the question remains: why hasn't (a) an FPGA vendor already done this and already actively selling a tensorflow solution or (b) NVidia pursuing this? I have a feeling there are more factors at work than just whether or not it's possible.

It's a very expensive, very customizable appliance. Hence Amazon's AWS F1 instance and Nimbix cloud.

Guess we need more assholes to write compilers

The problem isn't compilers but bitstream formats and the device description database (according to https://news.ycombinator.com/item?id=10653179)