Having two DDR controllers helps the overall memory bandwidth number if your reads are not localised. If you're doing a lot of data-dependent loads this doesn't help at all (e.g. scrypt).
In all cases it's much harder for software developers to develop for FPGA, so this cost needs to be factored in. They're very good in their niche, not a general purpose silver bullet.
Your example with scrypt also does not access much of RAM, especially with Salsa20/8 standard function. As far as I can see, it also has parallelization parameter, and computation within top loop can be done in parallel.
Yes, it is hard to program for FPGA. But not that much - I myself programmed a system that performed translation from (pretty much high level) imperative description of algoithm to synthesable Verilog/VHDL code. In a one and half of month.
In my opinion, programming for FPGA is very entertaining, especially if you do not write V* code by hand.
First, of course, are very efficient Ethernet ports and many other efficient hard blocks.
Second, you can fuse many operations into one. For example, you cannot fuse 3 FP adds into one operation nor on GPU neither on CPU. It is often 2 or 4, rarely something inbetween or outside of those. GPU and CPU operations on vectors can be wasteful as vectors can be underutilized and on FPGA you can create circuitry that fits problem rather well.
I also think that you are not quite right about I/O or memory boundness. In FPGA you can add another I/O controller and use extra device for I/O, loading spare resource in FPGA. Same for memory - DDR controller synthesized into cells won't be very optimal, but you can have many of them nevertheless.