That's how justice was developed and has always worked since day one.
Aside from hearing God voices, we don't have any other.
It's just that in our days we pretend that we have some "specialists" discussing the finer points.
But something without the people behind it it's not justice, just like something without a power to enforce it it's not law.
When somebody says "What the majority considers justice is not justice" they really only say "My arbitrary idea of justice is better than other people's".
As a cultural thing, justice is only what the culture (shaped by the people) considers as such.
(One could only argue about what's more useful (utilitarianism) but that also needs a commonly defined desired outcome and a somewhat agreed upon utility function).
For them it is. In Germany during the war what they did wasn't considered unjust (like in the South during legal slavery). And if the majority on earth agreed, it would be considered a-ok (like before countries started abolishing slavery, when it was just a normal thing for most peoples all over the world).
We do have some escape hatches from "it's just what most say it is", but those are based on religious and cultural ideas (including "scientific" conceptions of justice in the latter camp).
I think in the end: justice as an operative thing is what most people say it is (it can't be anything else). But there's also some common shared sense of justice, not well defined and not easily enforceable at all cases (e.g. one couldn't enforce it in WWII Germany), but which contains a core of the only universal notion of justice (don't cause harm without reason, don't kill, don't steal, etc).