I outlined what I did in another comment, here's a log from my current firmware: http://crna.cc/vpu_bootlog.txt

I think I'm on the right track but I don't have ARM working yet, most likely due to clock misconfiguration. Can probably fix it when I have more time.

Sidenote, I wish #raspberrypi-internals was more active :(