I find that to be one of the less troubling solutions to the Fermi paradox, to be honest. Try pondering the ones where one stealthy galactic superpredator has already taken over the galaxy, and is just waiting until we reach a certain level of technological advancement to exterminate us because it's currently not worth the trouble.
One troubling fact for example is that radio signals might disintegrate into noise after large periods of time, making detection via this method impossible.