Speaking from experience, the working out of a polite understanding does not work fine. All humans have practice at being human around other humans, and given a multi-year situation, they will have time and incentive to sort it out.

But not all humans have practice giving detailed instructions to temporary renters on how exactly to behave so as to be indistinguishable from the actual resident. The property owner's absence breaks the key feedback loop driving the polite understanding. And further, as I already explained, the temporary residents have much less incentive to follow whatever micromanagement has been ordered. I understand that you think it's the same in theory, but people are living not in your head but in actual apartments.

Of course, we are treating short-term rentals just like we are treating other home-based businesses: we are regulating them. Zoning laws are the general-case solution to this, and some cities are applying them here.