Even homepods get this wrong, and I'd be surprised if they only used bluetooth. I know they use wifi for presence (being on the same wifi is required for things like managing lists). Homepods also support UWB (you can bring your phone close to a homepod to transfer music playback to it, and it's used for setup). But I still get my living room homepod responding to me while I'm in the kitchen mere feet from the kitchen homepod with my phone in my hand (working on grocery list, for example).
Depending on the phone's proximity is still annoying, though. It seems weird that they don't seem to use voice proximity to determine which one is closer.
It's a hard problem. From my understanding, homepods use bluetooth to coordinate who is going to answer. Google Home, at least in my home, is terrible at this problem. Often devices in another room will answer.
Once someone has solved this problem, I'm on board!