Anecdotally, a Finnish friend told me that if you excel at one subject, you are encouraged by teachers to stop doing it, and instead spend more time learning the other subjects.
One problem is that Small-fish-in-big-pond-effect, a good chunk of us realized there that we weren't special anymore, there wasn't the honor and praise for being top- and consequently that drive was lost.
Plus, once some schools become well established and start to attract highly talented kids- kind of kids who can do well by themselves without much external help, the school itself, I mean teachers and administrators, doesn't have to be very good. Their success starts to depend more on their ability to attract talent, and less on actually nurturing and developing them. Past success bring more success- a sort of virtuous cycle.
One such disrupter can take away easilly 10 minutes of a class hour, and 1/3 of the class not understanding basics, means the teacher has to try again and again to teach them the things they should already know, while the rest of tle class stagnates.
Available AP classes/tests include STEM...so Physics, Chemistry, Calculus, Biology, etc.
The biggest difference wasn't separating kids into math classes and language classes, but genrally separating them by "good" and "bad". Having a random mix of students, where some can barely read, and others read two books per week (or some can barely multiply, while others code math problems for fun), is a very hard job for the professor, because in the end, they have to lower their standards, expectations, and the quantity of material until most students scrape by. Having a group of students that are good at "school stuff", means the teachers are having very little problems with the basics and can continue faster and prepare students better for university and life. If we had separation for stem-talented students, and language-talented and *-talented, it would be even better.
So basically... group better students together, and you can do a lot more with them, if you don't have to pull the 'bad' students behind. I am aware that this is bad for 'bad' students, and that it should be fixable somehow (they should have options to move to better schools), but good students thrive without idiots in class.