Most of the time you need only some of your tests (the ones you're working on currently). Then, preloading the framework becomes the real bottleneck. But it's solvable too, via spork (I believe, there is spork-testunit, but I never tried it).

Most of my coding is done in minute loops (add test / watch tests fail / add couple of lines / watch tests succeed). YMMV.