Another perspective: people are not indifferent to the set of ethics their community subscribes to. Ethics help form a common understanding about what it means to be a part of a community. Sure, morality is constructed and observable choices matter more. But morality is an aspect that separates humans from "beasts". Ditch morality and some get that much closer to considering atrocities -- e.g. enslavement of weaker groups, authoritarianism over entire populations -- because after all, "the universe is indifferent to your ethics [...]".

> why not a draw a line here, or over there, or instead way out there?

Do people who are sensitive to/ take into consideration the pains of weaker beings sound/feel like a pleasant group to be a part of? And yes, there are trade-offs.