This is my experience. The two houses that I share a backyard fence with are empty. A caretaker comes by once a month to have a look around one of the houses but otherwise they're unoccupied.