This is my biggest worry too. It's easy to turn on ASLR and PIE and SSP and whatnot and still have tons of obvious bugs in your actual business logic.

I'm sort of worried that these automated metrics are too easy to game, and that all it will teach companies do is to use different compiler settings without actually caring about, say, hashing their passwords or authenticating their cookies.