These are all legitimate concerns. In my experience, growing up in South Carolina (some of the worst public schools in the US) the opportunity to go to a "gifted" (I hate that word, takes all of the emphasis off of effort or any actionable quantity) school for high school probably saved my life [0].

What doesn't get brought up is how night and day being in an institution that challenges you surrounded by peers that interest you is (the classes we took were of greater depth than most of underclassman college course I later took). By high school I don't think worrying about late bloomers applies much and while I can certainly sympathize with wanting to do more to enable the lower performing schools and students, hamstringing the top performers can't leave you breaking even. And YMMV none of the people I've known have ever suffered from "looking down" on less intelligent people. Hell the opposite, being in a situation where you're challenged leaves you acutely aware of how little you actually know. We need more of these institutions moving forward, to make sure our young people don't fall into the cracks.

[0]https://en.wikipedia.org/wiki/South_Carolina_Governor's_Scho...