I think that the number of planets within a distance R is O(R^2), but the probability of an asteroid from such a planet reaching us is O(1/R^2). But if the probability of life on the asteroid surviving the trip decreases with a longer trip, the net effect is probably that the further-away planets contribute little to the probability of life reaching us in this way.
Similar to how ancient Romans wore silk but didn't have trade with China. It only took each trader to trade with their neighbor, and silk could travel thousands of miles.
The statistics is hard. No simple counting of planets is enough. All the vectors have to be accounted for.
You keep trying to say "but big numbers of planets, and big amounts of time" to make this idea reasonable. But it doesn't work. You have to deal with how low the probability actually is of it happening from one planet in one year. If that's low enough, you can't salvage it just by saying "billions of years and billions of planets".
There aren't even close to billions of other planets that could have been candidates for sending a life-bearing asteroid to Earth. Some of the restrictions are:
1. A planet capable of having developed life.
2. A planet close enough for an asteroid from it (I have trouble typing that because it's such an unlikely scenario) to have reached Earth around the time that life on Earth began. That was ~4 billion years ago, which seems like a lot, but consider that if the recently spotted asteroid 1I/2017 U1 had come direct to us from Proxima Centauri, the closest star, it would have taken at least 48,000 years to get here. That puts a limit on candidate sources of less than 100 times the distance to Proxima, and limits the number of star systems to ~2 million.
Even if you multiply the probabilities in question by a factor of 1e6 to 1e7, it's not going to change the conclusion.
We started with, it took an Earth to develop life, but if it came from elsewhere we'd have to expand our ideas to include all other possible sources of life, not just Earthlike. Because our premise changed.