You don't have to go through all possible addresses, you just need to find one that exists, no? Although my gut tells me that's not going to make much of a difference in terms of the size of the search space.
The point is that there is a staggeringly large amount of possible addresses, and that this space is incredibly sparsely populated by actual wallet addresses. You could generate an enormous amount of addresses looking for bitcoins, and not even come close to likely to finding any.
To put this into perspective, the universe is only approximately:
13.798±0.037×10e9 years old
Math is fun.