I suppose this all hinges on how you define 'fair'. As long as there is no agreement on that, it's hard to discuss this issue in a meaningful way.
Fair, from my perspective, is taking care of others, especially those weaker than me. I happen to have been born in a nurturing environment, with above-average intelligence, and lots of valuable experiences that inform my decisions. While I do work hard, it sometimes shocks me how many things 'came free' for me.
From my perspective, it is not wrong that I happen to have won the lottery. It is also not wrong that others have not, and have less than me. But I would say it is unfair for me to not share my advantage with others, and to try to increase happiness of those around me through this.
That said, I respect that your view of fairness is different. I just don't like it, and choose to be different, and I'll fight for my version of fair...
His logic is stupid at best, of course. You can practice climbing trees as an adult, and you'll probably progress significantly faster than you would as a child.
His response should be "No, I collected these myself, and you can collect yours yourself."