Your model sucks if you are arbitrarily choosing factors. You choose the factors with the most significant correlations, because those correlations are least likely to be "overly simplistic".

Why would we reject that zip codes are more likely to default on a loan? Seems like information I would like to be aware of if I was a home lender.

I certainly look at crime rates of a community before I live there. While a bad crime rate certainly doesn't make potential neighbors "guilty by association", it certainly increases the likelihood that one of my neighbors might be actually guilty.