As with the issuance of a warrant for wiretapping, there would need to be a proportionate and legitmate reason for your communications within a such a dataset to be looked at.
It's a bit like the police getting a search warrant to look around your home. If there's no legitimate reason to do it, like having reasonable suspicion of a crime that requires investigation, then they're not going to.
Yes, it is a bit like this. Except in this case the police don't need a warrant, they can enter your home for any reason at their discretion. You're putting a lot of trust in a bunch of people you've (I'm assuming) never met working for an agency that has demonstrated a complete lack of regard for the constitution. Either that or you're a really terrible glowie: "How do you do, fellow tech enthusiasts??"
And your edit seems to ignore that the analysts are humans. Police get caught abusing their access to data resources for personal gain frequently, why are NSA analysts different?
(Not even touching on the fact that mistakes happen, leaks happen, breaches happen, laws change, political winds change direction)
Let me put it this way: I don't do anything illegal in my bathroom, but damned if I want someone watching me in there. Everyone has their line they don't want crossed. Klein's - and the EFF's, and mine - is somewhere past the NSA monitoring every single communication in the entire country without a warrant. I have no objection with them monitoring specific suspects with a court order, but I don't want them listening to people who aren't being actively, personally investigated.