Right up until the guy says something disagreeble to a politician or federal law enforcement bully who just wants to ruin the guy's life. Allowing law enforcement to amass information that it "doesn't care about" does lead to abuse of power. See 'National Security Letters' for more information.