Since what you wrote is true (you could have a gene where one allele increases observed phenotypic variance and another decreases it; in fact I'm pretty sure such genes exists), I suspect the people downvoted you did so because they'd rather not think clearly about these matters.

It's not quite at the level of a single allele, but e.g. I understand that the same mutation related to eye-color results in a lot more color variance in females than it does in males.

And male vs female variance in general would appear to be strong evidence that something is going on.