Anybody with the number can join your Zoom meeting and since there's very little indication of their presence you might not even notice. You can also command other people's rooms to join your meeting, if you can figure out how to contact them, and you can force their microphones to turn on, which in my opinion is pretty disturbing.