Historically before the industrialization, being fair skinned was a proxy for being wealthy enough that you did not need to be in the field and work hard for a living.

After the industrialization, having a tan was a proxy for wealth. It showed that the person could afford being outside rather than working hard inside a factory all day.

Wealth and status gained from wealth is something which people inherently find interesting, and especially when there are changes. From a behavioral perspective, anytime people comment about someone tan I see it as a comment about status, similar to what happen if someone buys an expensive jewelry or car.