It's a bit like the police getting a search warrant to look around your home. If there's no legitimate reason to do it, like having reasonable suspicion of a crime that requires investigation, then they're not going to.
Yes, it is a bit like this. Except in this case the police don't need a warrant, they can enter your home for any reason at their discretion. You're putting a lot of trust in a bunch of people you've (I'm assuming) never met working for an agency that has demonstrated a complete lack of regard for the constitution. Either that or you're a really terrible glowie: "How do you do, fellow tech enthusiasts??"
And your edit seems to ignore that the analysts are humans. Police get caught abusing their access to data resources for personal gain frequently, why are NSA analysts different?
(Not even touching on the fact that mistakes happen, leaks happen, breaches happen, laws change, political winds change direction)
As with the issuance of a warrant for wiretapping, there would need to be a proportionate and legitmate reason for your communications within a such a dataset to be looked at.