They could legally move there, but they would be shunned from the community, and indeed it may be hard for them to find someone willing to even sell them a house.