I have a similar but marginally more lax policy: I'm fine with smart devices as long as they don't have to phone home to operate and they have on-device controls.
For obviously Internet of Shit-worthy appliances (Cory Doctorow's smart toaster or whatever) I'll always choose the dumb version, and I make sure that I only use home automation stuff that's as widely interoperable as possible, but otherwise I don't particularly fuss about it as long as it meets all of my other requirements.
I think modularity is really key here. As long as you can replace or update each component individually without impacting the rest of it, you should be good.
That's exactly it: the ability to easily swap out components, and then high levels of interoperability. I think I perhaps didn't express myself so well because I'm broadly in agreement with you. It's the phone home aspect that often makes smart devices a problem.
I tend to agree: I think switching it to a simple Bluetooth speaker is a reasonable way forward under the circumstances.
As a related aside this, along with a raft of other support withdrawals recently (the most recent that comes to mind is those smart glasses), I do feel somewhat vindicated in my policy to keep things as dumb as possible: speakers are just speakers, lightbulbs are just lightbulbs, my kettle is just a kettle, and - if I do need any smarts - I attach a general purpose computing device to provide them. E.g, it's just incredibly easy to connect my laptop or phone to my hi-fi via a DAC or audio interface to stream music over Spotify.
The point of this is that it means most of my household devices and appliances remain brick-proof, and I shouldn't have to keep replacing them every 3 - 5 years just because manufacturers stop supporting them.