I don't think it's that. The argument should be similar to the Veil of Ignorance: do you want every conceivable future government to have these powers?
Of course, the damage the bad apples do far outweighs the benefits of the good so that argument doesn't hold. A century of good governance with global assistance barely balanced out the damage Hitler triggered in 5 years. Destruction is easy and creation is hard. And that plays out on the small scale too with small numbers of corrupt officials undoing huge amounts of work by good people.
If we had perfect information, we'd find that the spying apparatus of the 5 Eyes has done far more bad than good to the general welfare. The political corruption that must be causing is immense, we already know that every president since Obama has been targeted by the intelligence apparatus in some form. It is unlikely it ends there.
I'm less sure about the situation in the UK, but they generally follow the US trend.
The fundamental problem is that given any permission (an extension of trust), individuals will--100% of the time--abuse it in the absence of oversight.
It's not a matter of if/when, only a matter of to what degree. But being individuals, we are empowered to choose how we grant or revoke such permissions to other individuals. If you stop trusting your spouse, you can revoke their access to your email.
Government is a superset of Individuals. There is no greater clique they are accountable to. Any permission extended (surrendered) to it is an implicit conversion of ownership. If you stop trusting your government to act responsibly with surveillance, you can't revoke that permission. It's no longer yours.
...like the toy you hand over to the playground bully because he "just wants to see it for a second." He was polite enough to state the conditions of release upfront-- but nothing on the playground compelled him to actually abide by his own terms. You find out too late he's a liar, and if you're lucky, you'll get your toy back, when he's done with it, and likely in pieces.
It's frustrating trying to protect children from amorphous threats whose access methods are abstract, virtual and obfuscated by incomprehensible technology and encryption. The question is always whether we maintain absolute anarchy or accept nanny-state dystopia.
Maybe it's time we consider kicking kids off the internet altogether before surrendering any more personal rights to a bully with the weight-class of an entire institution.
The entire rhetorical position is difficult. One of the (somewhat implicit) rules of public discourse seems to be "assume good faith". To make the argument that people should have freedom from government requires an acknowledgement that government officials regularly do more harm than good. It is insanely challenging to carry that position in a debate. Keep your eye out for the handful of people who manage to do it without being tarred as crazy. That rhetorical imbalance makes it safe and straightforward for power politicians and police to argue that they should get to read everyone's emails.
It is tricky to understand why centralised power is a terrible thing that will lead to corruption and injustice. Implementing a system that tolerates privacy requires the bureaucracy to not grasp power. We've always been lucky in a sense that a critical mass of people learned they need to fight to protect freedom as an ideal, even if they can't exactly explain why or if the details seem to lead to bad outcomes (see also "fight for your right to say it"). It may be the only defence against a Stalinist- or Maoist-style collapse.