Granted there could be some technical limitations that prevent you from observing certain systems but that pool will be much smaller than an entire galaxy of stars.
Also, don't forget that a Von Neumann probe is at a disadvantage when it arrives in a system since you want to send the minimum number of probes to each star and then have them replicate on arrival. If your probe accidentally shows up in a system that contains an advanced civilization capable of detecting and capturing it then you've given away the fact of your existence and your intent along with all the information that can be gained by studying the probe.
Also I'd expect the probes would communicate with the homeworld via relays (where they were replicated) rather than directly, so it'd take a fair amount of effort to unravel a replicator network unless you caught one of the very first probes.
And also it'd be conspicuous if there was a volume of space in your network where your probes always failed.
If I was also murderous, I would have programmed the probes to destroy any signs of life.
So, the fact that we are still alive after broadcasting the presence of life through out atmosphere for the last couple billion years is quite reassuring.