"Now, although you don’t see any Verilog or VHDL in the above tool flow,..."
http://forums.xilinx.com/t5/Xcell-Daily-Blog/SDSoC-developme...
Though Zynq is ARM+FPGA, in principle one should be able to do the same with a soft processor cum custom coprocessing logic in a fpga-only thing.
The limits of this is you're not making custom IP, but rather using a library of digital logic designs provided by Xilinx. Which works well for a lot of use cases, but it's not on the same abstraction level as actually describing the hardware itself.
My dream : FP-ish abstrations at any layer (so called hardware would become just another interpreter for a different level of data flow)
Registers, signals, etc. can change values, but you can't change what they do or how they're used. So you can't really have anything like self-modifying code. But I guess allowing state would make sense sometimes as long as you are mindful of what the state is dependent upon. Making lots of independent little state machines would map well to hardware I think.
C is a procedural language. Most languages are procedural to some degree, but C is entirely sequential. Digital hardware, unlike a C program, is entirely concurrent. When you write HDL for hardware, you're not really writing a program, you're describing the hardware layout itself. Really it's closer to making a logic gate or circuit diagram. HDL doesn't "run" like a software program. It just exists on the FPGA. That's the design, and it doesn't change once it's programmed on the chip. As such, the only "procedures" in hardware are from causality from something that happened in the past in the same hardware, either from a different part of the circuit, or the same part in a clocked design.
So what would be a good tool to describe hardware? I think (preferably immutable state) functional languages would be much closer, because they're much less procedural. You can more directly describe complex logic that takes some input, and gives some output, in a manner such that sequence of events does not matter. This would be much more powerful when mapped to an FPGA, because everything that didn't rely on a previous result could happen at the exact same time. And THAT much better utilizes the power of hardware. Implementing a long chain of sequential events on hardware is... not the smartest way to design hardware.