First, creating lesson plans/assignments (and grading them) is extremely time consuming. So it's completely reasonable for teachers protest having their workload increased.
Second, things like gifted classes and magnet schools create a "creaming" effect, where the best students are pulled from regular classes and either sent to a separate school or in separate classes with high performers. If you're a math or English teacher that's getting professionally evaluated based on your students' performance on standardized tests, you are strongly incentivized to keep any high performing student in your class - the existence of gifted classes and magnets hurts you.
In the tech world, we sort of had a version of this during the Ballmer era of Microsoft, where engineering managers were sometimes reluctant to grade top performers too highly in fear their best developers will get reallocated. Some managers would give elite team members bad rankings just to keep them captive.
Teachers can simply be evaluated based on how their students do, relative to the students' prior performance. This wasn't a problem before 2014, when CA made GATE optional. Everyone understood that different teachers had different cohorts with different average abilities.
The teachers with lower-performing students weren't dinged because their students scored lower than the students in advanced classes. Instead, teachers could specialize in remedial, typical, or advanced learning.
IME the teacher unions are also opposed to advanced learning. It requires teachers to do more work if classes are heterogeneous, since they would need to make up more lesson plans and assignments. And the teacher unions are opposed to tracking because then teachers don’t all have the same mix of students, which leads to member complaints. At least this is what we’ve been told.