It's because of the historical, and thus causal nature of the distribution of opportunity that we might have concern over the distribution.
But if we re-rolled the dice, it's not like the asian couple from around the world is suddenly going to have a caucasian baby. You being born where you were was a confluence of factors operating in a causal framework with some noise. A re-roll isn't going to make you a Hilton family baby.
Also, what a person is entitled to is a matter for society to determine. Society creates a construct that it feels is a wholesome lens or boundary, then it creates the structures to enforce it. There is no natural right to anything, including property, life, or even death (some states forbid you to choose your death). Society determines that property is useful, creates a structure to enforce it, and voila, you are now entitled to property.
And you claim that "My family history made it happen" and "I do not wish that everyone in this world can have what I have"?
Wow.
You must have a really interesting family tree.
I don't necessarily agree with that logic myself, but then I also don't feel that I'm entitled to what my 'heritage' brought me.
Personally my main concern with this whole issue is practicality. I'm concerned that a too-large influx of refugees (or economic migrants) might cause severe destabilization and further increase the growing right-wing sentiment.