How are you getting cheated if it makes no difference?
Because you miss out on the chance to get angry?
As other have mentioned the inferior oils might not be as healthy, this might be a reason to care not just 'cause'.
As I just amended my comment up-thread to say: allergies. Food allergies can kill you, and nut oils (nuts being a fairly common, and often severe allergen) are sometimes used used as an olive oil adulterant.
Adulterated olive oil isn't even labeled that it's adulterated, let alone what with.
So, "Fuck 'em if they don't know enough to know they're getting cheated?" Nice.