And to add onto your point there's a lot of nuance in terms of what it means to be against illegal immigration. Some people define it very harshly (toss out everyone that's an immigrant), some people define it as providing avenues for hard-working undocumented immigrants to become US citizens even if they are illegal.

So when you come out against illegal immigration, your stance can run the gamut of not racist at all to incredibly racist. It just turns out that most of the time the statistics simply don't line up for illegal immigrants committing more crime, stealing jobs and so forth. Which then makes the argument not about fact, but about feeling. Which then tends to be more inclined towards racism.