There is software and hardware involved in encoding/decoding the analogue signal, and compressing it for transmission across a digital network. That involves automated processing, without which the system can't function.
I'm very much opposed to surveillance and other forms of privacy invasion, but I think the discussion needs to be based on a clear understanding of the nature of the processing involved, and the debate to be around which forms of automatic processing fall into the "acceptable" and "unacceptable" category.
Examples of the latter, in my view, would include storing a copy of your content for longer than necessary to transmit the call, doing speech-to-text analysis and using the extracted words to build up a profile on the caller/callee, and making the recorded information available to government agencies for surveillance purposes or corporate entities for advertising purposes.
"Listening in" is an insufficiently-defined term, which doesn't provide enough information for us to have a proper debate about exactly what's going on.
Whether or not Google should be allowed to perform additional processing of the emails in order to choose appropriate advertising is a separate issue. I'm concerned that a lot of articles are missing this critical distinction. The focus of the debate needs to be whether they can deliver advertising which is based wholly or partially on the content of your emails; they fundamentally have to (electronically) read your emails in order to deliver them to recipients.