I occasionally use the omniscient debugger to debug some particularly tricky bugs, and it's always proved quite useful.

There is also a more modern-looking, commercial implementation of the same concept, but I've never used it: http://www.chrononsystems.com/