The problem is that it's really hard to write unit tests to determine if you've gotten the crypto right. For one, the unit tests would have to have a standardized interface to your cryptographic protocol. But if you're using a standard cryptographic protocol, why would you be rolling your own implementation instead of using an existing, well tested and vetted one?

Second, it's quite difficult, in an automated way, to look for the kinds of security vulnerabilities you would need to audit. Timing attacks may show up in different places for different software. Poorly seeded entropy is quite hard to detect; Debian systems were generating keys with only about 16 bits of entropy for years, which is enough that it's hard to detect in an automated way unless you are really looking for that specific breakage but is a devastating security hole. You may have bugs that are not crypto related at all but simple buffer overflows or the like. And so on.

You could probably automate some of it. But on the whole, you need many good, experienced people regularly auditing the code. And it's a lot easier to get that by using an existing, widely used and widely audited library than by rolling your own.

However these are just statistical tests; they tell you nothing about how much actual entropy you're actually getting (cf. Debian). That said, it probably would have caught the Cryptocat bias.

I'm not actually sure what "actual entropy" would mean in this context if not "the number of bits in the seed".

There are many things that can go wrong. Some examples:

- the seed is timer-based and has far less entropy (information) than its bit size

- the seed is OK, but the generator is somehow weak (e.g., only made with linear components, or throws away parts of the seed, or outputs too much of its internal state)

- Bugs, bugs everywhere (cute example: an RC4-based generator that uses the XOR trick to swap elements)

No standard statistical test would catch those (it would catch the latter, actually), even though the security of the generator would be gravely diminished.

Also, a lot of the time the vulnerabilities are not in primitives like PRNGs, but in how application programmers combine the primitives in their application, and there's no way to unit test the security of that. The way to mitigate this is by developing and using high level library interfaces that make it very difficult to screw up.

At the moment, the only "correct" test I know of is hilariously impractical: enumerate all possible programs up to a reasonable length, run those programs against output from the random number generator for a few minutes each, and see if they're better than chance at recovering the seed you used.