You speak as if the problem is that it could be the security apparatus has been abused, but we don't have any evidence this has happened.

The problem is we cannot be sure exactly when it has been abused. It's only fair to presume our our corrupt system does not use the NSA ethically.