If it's racist to treat people differently in a way that affects people of some races, than anything less than open borders (allowing anyone to immigrate for any reason) is racist. If you don't support open boarders, you're either a racist, or that definition of racism is too broad.