> A single tree might make hundreds or thousands of seeds each year but the number that sprout and survive to adulthood is a very small fraction, there is a lot that needs to go right.

While this is true, I think you are stretching this to fit your particular narrative. Seedlings that sprout from a nut that "didn't fall far from the tree" have a lot of things going against it. The shade of the mature trees around it prevent it from getting enough sunlight. There are other factors that contribute as well.