You only have to miss the right time period by several thousand years to miss signs of (intelligent) life.
Someone remotely viewing earth with just a 10k year delay would not be able to measure signs of human life.
They would see exactly what we see elsewhere - nice possibly life-supporting planets, but no signs of any functionality.
Now realize we aren't just missing 10k years but 1000k years (at best).
EDIT: Actually, I missed your point, which was that far away things have less history in which intelligence could evolve. This is true, but the amount of history we can observe is still staggeringly large. Far too big to eliminate the paradox.