Yeah, they're a bit of a dark art. They require hardware design tricks like pipelining for best performance, you can run into fun stuff like routing fabric congestion which absolutely kills performance and in turn is affected by stuff like the layout the tools choose and whether you're using cheaper or more expensive FPGAs (which at least for Xilinx have very different balances of routing resources), etc. I've seen timing failures due to the signals having to divert around the clock generators at the edge of the chip, which were effectively a wedge with no routing fabric in between two sections of logic.