Some of it may be crab mentality. No matter how much you spend on some (but not all) special ed kids, there's no worry they will academically surpass the majority of students. So there's no fear from parents that the spending will mean their own kid gets dragged further down the pecking order.
Spending for gifted children means putting other kids even higher up the pecking order than their own average kid. Who wants to spend a bunch of money to watch their kid get left further behind the front of the pack?
To be fair the author was fairly balanced and presented the arguments against, such as that they tend to come from wealthier families that can provide that support, that they will be fine on their own without it whereas the bottom 1% need the support, and so on.
But it was an interesting thought experiment none the less. What would our society look like if we spent as much on the top 1% of students as the bottom? Or do we already via college education?