That means making a whole networked black-box, based on FPGA(s) and exposing an API for external use.

It's certainly possible to make. It's also a very expensive very specialized appliance. All of that to do some matrix manipulations.

People are buying large quantities of very expensive GPUs just to do some matrix manipulations, so why not FPGAs?

But the point is that you don't have that much FPGA-specific code - once someone does the matrix manipulations and the proper integration, everyone else can just run e.g. tensorflow code on it faster and/or cheaper without specific expertise; if a FPGA vendor can do this one-time investment in software tools, then they can compete for a slice of the large pie of hardware revenue that nvidia now has for itself.

But, assuming that "matrix manipulation" code has existed for FPGAs for a while, since verilog and VHDL are quite old, the question remains: why hasn't (a) an FPGA vendor already done this and already actively selling a tensorflow solution or (b) NVidia pursuing this? I have a feeling there are more factors at work than just whether or not it's possible.

It's a very expensive, very customizable appliance. Hence Amazon's AWS F1 instance and Nimbix cloud.