It depends on the area. Most go back to when the area was developed, some of them were voted into an association (eg historic heights), others are simply drawn at class lines (river oaks is entirely an association). The lines are really arbitrary. My house has a very minimal neighborhood restriction (no commercial front businesses) while my neighbors a few blocks away are required to maintain sidewalks, upkeep of yard, etc. Condos and townhomes obviously have HOAs as well. I believe a complete tear down is the only way to part ways with a deed restriction (which is why historic homes being remodeled are sometimes razed in the middle of the night)