Government agenties are more likely to, for example, expand on publicly available research that uses malicious software and audio I/O built into (or connected to) computers to remotely access air gapped computers.[1] And, if it's possible to write a network protocol that runs over low quality mics and speakers up to 65ft apart, it doesn't sound unreasonable to similarly hack up audio drivers such that a speaker can function as a microphone for the purposes of espionage, without anyone breaking down your door or rewiring anything. Spies don't have to convince a jury with the intelligence/evidence they gather, so low quality can often be good enough.
Of course, an attempt to aggregate the compromising of countless bespoke proprietary audio drivers and operating system versions doesn't scale at all... Which is why I used the wording when surveilling a target and not when surveilling speakers everywhere. That latter claim would indeed be paranoid nonsense. Which is why I didn't make it.
As in, a government is surveilling a particular person or building, has compromised this a nearby computer remotely, but the computer has no microphone. If there's a connected speaker (or even an integrated speaker on the motherboard), deliver a payload that messes with the audio drivers to see what you can hear using the speaker.
[1] http://www.jocm.us/uploadfile/2013/1125/20131125103803901.pd...
I don't doubt there are people working for most major governments' intelligence departments who can use malicious software to facilitate listening through a computer connected speaker when surveilling a target (even if that speaker doesn't have an integrated microphone, in a very literal sense it is a microphone).
[1] https://security.stackexchange.com/questions/154343/can-a-sp...
Edit: precision of language