Lumping totally dis-similar arguments together is just intellectually lazy, which is most of my problem with the use of words like "racist, sexist, etc." They have valid applications, but are often used to circumvent rational debate.

Most people are intellectually lazy. Better get used to it. If you want your arguments to be consistently distinguished from the racist ones you will have to put in more effort to make them seem more distinct.

You don't even know what my arguments would be!

And to add onto your point there's a lot of nuance in terms of what it means to be against illegal immigration. Some people define it very harshly (toss out everyone that's an immigrant), some people define it as providing avenues for hard-working undocumented immigrants to become US citizens even if they are illegal.

So when you come out against illegal immigration, your stance can run the gamut of not racist at all to incredibly racist. It just turns out that most of the time the statistics simply don't line up for illegal immigrants committing more crime, stealing jobs and so forth. Which then makes the argument not about fact, but about feeling. Which then tends to be more inclined towards racism.