Or maybe a compiler could recognize that optimization is possible and create it for you

More likely a JIT can do that.

The FPGA can't be reprogrammed fast enough for the JIT approach to work, unless you're running computations that take many minutes or hours. I suppose that does apply to some workloads, but it would be tough to ask your JIT to solve the halting problem and guess whether a workload will last 10 seconds or take longer than that.

For a JIT, the easy way to guess is to wait until it's already taken 10 seconds and if it hasn't stopped, assume it will take at least 10 more.

I would say that program that are run and exit immediatly are a minority of what is produced with languages. A lot of web server, services, gui are produced in JIT languages and have a timespan of multiple minutes. AFAIK FPGA reprogrammation time depends on the size of your edit, and your hardware, the article says that they expect to reprogramm it on a program load, so I don't think it will be that slow.

Why can't a JIT program the FPGA based upon previous runs through the same set of code? IIRC the JVM won't rest on its final optimizations until it runs a chunk of code hundreds (or thousands? I forget) of times.

Depends pretty much on which JVM we are talking about, Java is like C, plenty to chose from.

In what concerns OpenJDK that is configurable via the -XX flags.

In scientific computing, that is the typical workload. That is why, say, Julia exists despite having a ridiculous JIT overhead.

Isn't that a general warm-up problem with JITs though?

With any kind of compilation process, a balancing act between time spend waiting and the quality of generated machine code.

I've been doing that for years. I have created custom microcoded CPU's in FPGA's for tasks where it would provide an advantage. One example I remember was a microcoded real time image warping engine.