I think it's useful to substitute athletic talent for intellectual talent when thinking about this stuff. Does anybody discount the value of getting an athletically talented kid into a school with a good athletics program where they can be around peers and coaches who will enable them to be the best athlete they can be? Would anybody argue in favor of letting some kid with the athletic talent to try for the NBA flounder at whatever high school they happen to naturally feed into?
I think the answers to those questions are obvious. And yet we regularly have people arguing that somehow the same principles don't apply to intellectual talent. It's ridiculous. Let the mental prodigies be with their peers. It hurts no one and does them lots of good.
It's true that I was better at school than my peers, the pace of teaching was too slow for me and I got bored easily. But it was also true that in other aspects, especially social aspects, I wasn't more mature than my peers. And being smart and having different interests (say, computers, space or philosophy instead of football, etc.) doesn't exactly help socializing at school. I was always relatively unpopular (not marginalized, fortunately, as I did have my group of friends, but I did get my dose of demeaning from many others) and not very good in my advances with the opposite sex (when the age came to be interested in that).
Altogether, I think my school years could be called rather OK (taking into account that, as mentioned, people with "nerd" interests tend to not be too well-liked). But if I had skipped a year, and I had been in class with people one year older, with more social maturity, with everyone knowing that I skipped course, the girls seeing me as too young to even think about considering... I think it would have been hell.
I'm aware it's not exactly what you're talking about. You imply a specific school for gifted kids. Still, I think it wouldn't have been a good idea. When you finish school, or university, there's a point where you have to interact socially with "normal" people. Both at work and social life. I learned those interaction skills at school. I learned, for example, to fake interest when people talked about what I deemed to be trivial maters, like football, TV shows and the likes (or later, even to have some genuine interest and get off my high horse sometimes). I learned when I should switch off nerd mode (for example when trying to interact with the opposite sex, especially at the beginning) and when I could actually benefit from it. I think those skills were vital for my well-being. And I doubt I could have learned them equally well at a school where I weren't able to socialize with "normal" people.
You might find this study interesting. https://www.davidsongifted.org/search-database/entry/a10489
Second, non-athletic education and interests get lost in the process. And again, most kids wont end with that career as there are only few slots and many competing kids.