That's more of a problem with HOAs than with townhouses. You don't have to have an HOA to have townhouses and detached houses sometimes have HOAs.

I have a non-HOA townhouse, and there's basically nothing I can't do with it that I could do with a detached house short of knocking down one of the side walls.