Most law enforcement agencies argue (with varying degrees of believability) that they will not treat an encryption backdoor like a universal bug they can just listen in on whenever.
When you get a Judge to agree that their is adequate reason and gain a warrant, you use your resources to surveil the target in public/private, using your considerable means as a law enforcement agency. Drones, bugs, malware, other espionage techniques.
This is nothing more or less than these agencies complaining that free peoples are making their jobs harder.
One big problem is cowardice: it is easier and safer to attack the liberties of law abiding citizens (because they won't shoot you in the face), in the hopes that, by sacrificing your citizen's liberties you might pick up a few more incompetent bad guys. Its way more dangerous to actually go out, do your job, and surveil a dangerous bad guy, who will shoot you in the face.
If that's a problem -- then the law enforcement complainers need to go get a new, safer job, and leave the heavy lifting to their braver counterparts.
You don't have a general right to keep secrets from law enforcement. With a warrant, they can spy on you to try and get your secrets in ways that would otherwise be scandalous or downright illegal. If you don't think law enforcement should be allowed to steal people's secrets at all, fair enough, but that's a very radical view and I don't think most people supporting E2E encryption share it.
More to your point, I do think law enforcement should be allowed to spy on people when necessary (by whatever definition of 'necessary'). But that's a far cry from saying that people should be compelled to assist in the spying.
There's no warrant to compel someone to knowingly carry a bug.
They were also never able to learn what was said in the van before the London Bridge attack [1]. Are we okay with Renault providing a law-free space inside their vehicles? All cars should be equipped with always-on microphones. But don't worry! The audio will be encrypted, and only accessible with a warrant.
[1] https://en.wikipedia.org/wiki/2017_London_Bridge_attack
Edit: Just to make sure - this is satire, and the logical extension of believing people shouldn't be allowed to keep any secrets.
Responding in edit due to posting limit (way to make sure downvoted comments can't explain or defend themselves):
They can also record the person typing in their password, or recording the screen with cameras, or hardware or software bugs once they have a password to access the phone.
The analogy works perfectly, and while they may say the will not "treat an encryption backdoor like a universal bug", that's exactly what it is.