But even when they have a warrant to go after one person, the fact that they can scoop up information from everyone else even remotely near that person, and those people never find out is pretty abusive.

Like, if my neighbor is a drug dealer, and the police get a warrant to search their apartment, maybe that's reasonable. If they use that warrant to search every apartment on my block, that's messed up. If they do that without leaving a trace, so I have no idea when or if it has happened ... then I'm living in a panopticon.