The notion that gifted students need a special environment to learn is comical. If they're actually talented and gifted, they should certainly be able to learn effectively in the same environment as "normal" students.
Learning to read and do arithmetic is important in the 3rd grade, but it isn't as important as learning social and communication skills. I don't deny that there are gifted individuals among us, I'm advocating that we wait on trying to identify the gifted ones at least until puberty.
Another data point from a personal perspective: I was fairly gifted (though far less than the names I mentioned) and came up with the idea of limits in math on my own when I was 9 years old. I never got any proper mentorship and the development of my math skills and intuition was held back by the boring classroom materials and routine homework I needed to attend/finish. I did not have access to or even aware of any advanced materials I could learn from, and thus wasted several of the best years to develop those math skills and intuition.
They may need a special environment to be allowed to excel in learning without getting bullied.
You're thinking about this as if they're taking your resources. Let me turn this around for you. What if they aren't being given special treatment because they need the resources, but because they need an environment where the way they think is acceptable? How many of them would still show any talent or aptitude in normal class with normal classmates? Think about your reaction to negative numbers versus their own. Maybe gifted and talented classes aren't so much giving them more resources as they are giving them an appropriate education. Maybe it's about creating a safe environment for kids who tend to be more abstract to flourish.