Because then he could be married before he was even born? Or a young person could "feel" like you are 65+, and start collecting retirement benefits early? Or they could "feel" 21+, so they could buy alcohol? Or if by changing your age, then a crime you committed when you were 18 was retroactively changed to when you were a minor?

Changing your gender doesn't actually have many legal implications (outside of, perhaps, marriage). Changing your age would throw your entire legal history into question.