They're in the Nasdaq and S&P 500. Even if you don't own them they do have systemic impact on the rest of the market, given that lots of money tracks those indices and they're used as a measure of economic health.

Not investing in banks didn't make people safe from the financial crisis.