Additionally a user is likely to pay a lot more attention to their phone than to their Nest devices. A compromised Nest device will likely stay compromised until Google find the exploit...

Are you sure about that battery drain?

A malicious actor could easily conceal their activity by making 24-hour-long recordings and sending them in the night (or whenever connected to WiFi and plugged into power).

The main trick smartphones use to have their battery last long enough, is to power off every piece of hardware that's not in use, for as long as possible. Doing a 24-hour-long recording would require the main CPU to be awake far more often than usual (and in fact, I would suspect it would have to be pretty much constantly awake, unless the phone had a large dedicated hardware buffer for the recorded audio samples).