Several political philosophers critical of liberalism (and even those sympathetic to liberalism) have argued that several free actions we take do indeed disadvantage people in various ways, whether it comes to democritic citizenship or other social interactions. It is entirely possible to act under the flag of liberty while still (knowingly or not) disadvantaging other people. This 'paradox' (speaking loosely) has given birth to the (currently popular) philosophy of egalitarian liberalism (or, liberal egalitarianism, as opposed to other theories such as Marxian egalitarianism).

You can read more about this at the Stanford Encyclopedia of Philosophy: https://plato.stanford.edu/entries/liberalism/#NewLib