Those two modes aren't quite good enough, though. One recent fun episode occurred when I walked up to our house and started doing some work outside, while both I and someone inside were listening to music. For some reason, my phone automatically disconnected from my headphones, and connected to the bluetooth speaker inside, somehow disconnecting whatever it was originally connected to. So, for the person inside, the bluetooth speaker stopped playing their music and started playing something random, which proceeded to start and stop a few times, and then get louder and louder and LOUDER while I was standing outside, fiddling with my phone and headphones, trying to figure out why my music stopped playing.
My guess is that the root cause is that there's a crippling design flaw at the root of the Bluetooth spec: Its original designers apparently never considered the possibility that multiple people might share their electronics.
A better solution would be more robust cross platform Bluetooth stacks. We have tons of high quality, open source TCP/IP networking software, elliptic curve cryptography and key exchange libraries. The same needs to be done for Bluetooth. If companies like Google have enough money to relaunch and rebrand IoT home assistants every other week and publish new networking stacks written in Go for their latest container orchestrator/mobile OS/browser every 6 months then I am sure they can find time and funding for better Bluetooth firmware.