I understand your point, but despite the fact that the Y chromosome does not carry that much genetic information, it is highly susceptible to genetic drift. A smaller male population passing on its genes would cause a higher amount of genetic drift (a smaller population means that outliers have a larger effect on the genetic makeup the population), which would cause the Y chromosome to change at a faster rate (and it is already much more susceptible to change than the X chromosome).