Baseless in the same way as me saying "I bet your mom dropped you a lot when you were a baby." It's a claim not based on anything and therefore baseless. It's racist because you are claiming that race is a significant factor.