I have similar stories for every single bluetooth device that I have ever tried, which amounts to probably a dozen peripherals and half that many hosts. I've had fleeting moments of "it's magic" to punctuate the sea of crap so I'm fully on board with the potential of the technology, but it has continued to mature at an obnoxiously slow rate.
It gets wonky if that single speaker has paired with multiple devices before. For example in a home, where you've paired it with your laptop, phone, iPad, etc. Try it out. Audio will cut off sometimes suddenly when it decides to switch pairing partners as another paired device appears in range.
To connect to a different one, hold the button longer and press connect on the other device.
I use it on my iPhone, Android tablet and Windows 10 PC
So I have an iPad, macbook, and iPhone in my room all turned on at the same time. All with bluetooth on. All have been paired with the speaker.
I now turn on the bluetooth speaker.
How does it decide which device to connect to? Amazingly, it is always the wrong one. Every single time.