The racism is in the context of deprivation. Watermelon in and of itself is neutral. In America however, freed slaves did not have access to good agricultural land that could grow many fruits and vegetables. Watermelon grows on poor quality soil, so it was available to the poorest black sharecropper.

Because of this, it became associated with black people, and thereby with criminality, low education and poverty among the dominant culture that created this sort of pattern-matching.