Sure, if there's serious interest in you, there might well be high tech listening devices, lasers watching vibrations on glass windows, bugs, etc. But that's expensive, equipment, people in white vans, etc.

But what if the government can listen to every whisper in every ear? Dragnet surveillance could serious damage society. If this was available decades ago it might well be used to criminalize things like being gay.

So then why do they need weak encryption? Just do the same thing to capture the data at the endpoints, or use those methods to get the user's passphrase.

You cannot rewind your speech to hear what you wispered two years ago or do a quick search to see everyone you have ever wispered to. Without secure encryption the collection and storage of data by the NSA will give them those capabilities on data. So in your scenario it would be like always having a mic pointed at you, always being followed by wired informants and having everything around you bugged.