Note for cryptographers/matematicians: I know that "reversing" isn't the correct term here, so you could accuse me of the same sin I'm calling out in my previous comment. But it makes the explanation shorter while still conveying the correct meaning in the end.
Having the public key is easier than having an address because an address is the hash of a public key. So in order to crack an address, you must first find a public key that produces that address, and then find a private key corresponding to the public key.
Sure, finding a private key whose public key's hash is given might be 2x slower, but Pollard's rho algorithm is 2^128 times faster.
Asymptotically, an additional hash at the end doesn't matter when you brute force. But it prevents you from using Pollard's rho algorithm, which does make a difference asymptotically.