Yes, that's what is being implied, and it's certainly something that happens, in degrees. People generally enjoy attention from those they themselves find attractive - obviously up to a limit, but that limit is much different than when you don't find the other person attractive.

And that is why the "making a woman feel uncomfortable constitutes sexual harassment" standard is problematic. And of course it's not entirely wrong either because it certainly can be done on purpose.