On some theoretical level the HOA contracts / rules are voluntarily entered into. So even the most hard-core libertarians (or exactly those!) should be into them.

Of course, in practice you often enter a HOA as the lesser of multiple evils.