You're right - any conversation at people's desks in an open plan office should essentially be considered open for anyone to listen and join in. Otherwise it's just plain rude or inappropriate.