This is precisely the problem: making it seem like everything is "fair" because all kids get the same lowest-common-denominator/no-child-left-behind education system. Every child deserves to have the opportunity to learn, which is why the special education system (which receives enormous resources, far beyond the proportion of students who receive SPED services) exists. According to your logic, we could just get rid of SPED and say "it's all fine because all children have access to the same public education". But that misses the point, which is that different children need different resources to learn. Some need remedial support, and some need more advanced learning.

And as mentioned upthread, since CA got rid of the requirement for GATE education, schools (including those in incredibly well-heeled districts) have gotten rid of advanced learning options for students. The result is that kids with involved parents procure outside resources, while their peers whose parents are not involved/as well-off are robbed of the chance to learn.