This was featured recently in Bryan O'Sullivan's "Running a startup on Haskell" presentation: "[QuickCheck] generates random data, then tells me if all properties that I expect to be true of my functions and data hold. QuickCheck is shockingly more effective at finding bugs than unit tests."
There are ports of QuickCheck to tons of other languages...
For example, MS recently released a library for Design by Contract (aka Contract Programming) in .NET 4.0. It allows you to specify constraints on your methods, such as things you expect to be true before the method is called, after, and between all method calls in a class. The library is capable of static verification, but it's partly an experimental feature and you need to pay for expensive version of VS. (Runtime checking is available for free.)
But, here is the cool part: MS also released an automatic test generator called PEX. It can do exactly what you've described - go into your code and automatically find edge cases, and generate tests that cover each of them. And it's free.
So, you can write contracts, run PEX, and if something goes wrong, you will see which inputs generated exceptions.
D also has DbC functionality. I don't think it runs any static verification on it yet, but you can use it to detect abnormalities during functional testing.
It may seem a bit verbose, but it usually expresses logic that would be in your program or in your tests anyway. Difference is, you will be doing it in a pretty terse and declarative manner. IMO, DbC is one of the coolest features of .NET 4.0.
If you really do need tens of thousands of tests because you have so much interaction with unique states going on in your engine then you will probably never ship a product with any reasonable quality.
And that floating point operations lacks precision is just the way things are, and you need to understand how to write stable code despite it if you want a stable graphics engine.
This occurs so often and with such regularity I am now convinced that everything I write is riddled with bugs that I will probably never find without beta-testing in real-world scenarios, plus many more bugs I will simply never find because they never come up.
A much better design would be a test platform that analyzes the assembly structure to pinpoint actual edge cases in the code itself, which could then be used as a guide for finding bugs instead of relying on hundreds of thousands of test cases.