There is a specific reason to say it: 98% of polish population belong to a single ethic group, and even of the remaining 2% the vast majority are what most people would consider to be "white". Non-white people tend to be very rare even in metropolitan areas (maybe to a lesser extent in Warsaw), and they are usually either students or high-income people moving to Poland for professional or relationship reasons. In general, Poland isn't a very tempting target for immigration.

This said, you can argue there is some systematic racism against the Roma or Ukrainians. But it's more based on culture and national identity, so its different than what an American would think of then hearing the term.