The reality is that wealth and "privilege" are a scale: no matter who you are in the world, there are both people above and below you, in terms of advantages.
Even then someone would say that it was still done from a position of privilege because you had a place to sleep, animals to hunt, and how dare you say you are self made - you at least had a job unlike many people in this fine country!
To me, "self-made" means being able to take the opportunities within the environment that you live in and make the best out of them. Yes, that usually means that you get some opportunities that other people don't get - but equally, there are opportunities that you don't get either. You can be a "self-made" white son of a billionaire. Or a "self-made" immigrant of colour. Alternatively, we can stop using the term "self-made" entirely and pretend that it doesn't exist. That's also a solution.
Self made wealth is almost always a product of an entire community.