That's basically what John Rawls ends up concluding [1]. But the same line of thinking also leads to the conclusion that Free Will doesn't really exist, rather we humans are a product of our environment/genetics (and thus don't inherently deserve better treatment or rewards since all our efforts and subsequent results aren't 'really' our own doing). I wrote a little bit about this here [2]

[1] https://en.wikipedia.org/wiki/Veil_of_ignorance

[2] https://www.hugomontenegro.com/blog/the-absurdity-of-free-wi...