My understanding is that special ed money is distributed according to a power law: most kids classified as "special ed" actually need very little help (and very little money), and don't disrupt things for other students. I had a friend in my middle school Latin classes who was blind, and while he needed special written materials and some private tutoring, he didn't need any other help. The same is true of people with dyslexia, people who are wheelchair-bound, etc. They need some accommodations, but they are not where most of the money is spent.

Conversely, the kids who do need tons of resources are usually kids with severe mental or developmental disabilities. These kids usually have a 1:1 aide telling them what to do and trying to help them either understand the lesson or work through a totally different lesson (which also must be a humiliating experience - I would never want that for my child). I have seen both of these cases in public schools. These kids would certainly be better served by having a teacher who can pay attention to their needs instead of a teacher who can't and an aide who tries to keep up.