I'm not trying to defend the guy if he actually meant what people here seem to think he meant, but it's possible he was referring to how society ('we') doesn't care about minorities, rather than saying that he himself doesn't care about minorities.