This is a completely reasonable argument. You should absolutely have recourse if your neighbors (short-term or long-term) create a disturbance.

On the other hand, if they behave themselves just as well as long-term owners, then no, you really shouldn't have any say whatsoever about who lives in your neighbor's property.