VPU is not a GPU, it was designed for stuff like video decoding. It is just a simple RISC with some SIMD instructions.

QPU is a different beast, in VC4 it cannot even run arbitraty C code. In VC5 it can, but inefficiently.

Yes and no. You can definitely run arbitrary C code on AMD's GCN ISA if you're so inclined (and I expect every other modern GPU, but I know less about those...). There are all the usual assembly instructions, there are pointers, you can implement a stack, and so on. That doesn't mean that arbitrary C code will run fast :)

To fully use the computational power of the GPU, you have to make use of its parallelism. That means dealing with the fact that you can have hundreds or thousands of "waves" (things with a register file and a program counter) in flight simultaneously, and each "wave" corresponds to many (in AMD's case, 64) threads in the conventional sense.

It is the last part that makes the biggest difference compared to regular CPUs, because it changes how you have to think about control flow. If/else-statements must be compiled in such a way that the wave goes through both branches if the threads in the wave branch differently (if all threads branch the same way, you can of course skip the other branch).

The first part makes a big difference as well, of course. GPUs care far less about single-threaded performance, so there is no out-of-order or speculative execution, and the memory latency is high. When a wave has to wait, the latency is made up for by scheduling another wave instead. That is, there is a high level of what is called "hyper-threading" on the CPU.

They do. Or, at least they did. But, their capabilities have been improving rapidly for a couple decades now. To the point that shader compilers are integrating recent C++ features and some degrees of straight-up C++ support. You won't be able to magically run classic, single-threaded programs fast. But, you can start using advanced features you already know from CPU programming to write programs for the GPU.

Yup. Check out the VideoCore ISA:

http://www.broadcom.com/docs/support/videocore/VideoCoreIV-A...

Note that in this case, the author is writing the bootloader firmware so performance isn't a major concern, though.

This isn't the same, the document in question relates to the QPU ISA. The VPU ISA hasn't been officially documented but there were many projects that involved reverse engineering it. The VC4 ISA is documented here:

https://github.com/hermanhermitage/videocoreiv

The VPU is basically a general purpose RISC processor with some fancy vector instructions on top. In fact, most of the firmware that runs on it is written in C.

Ah okay, this explains a lot! I somehow thought that the initial boot happened on the QPU.