I think people draw the distinction because there's a huge difference between finding someone sexually attractive, and sexually assaulting them. (This applies to attraction to adults, not just children. "I find that person attractive" is very different from "I'm going to have sexual contact with that person without their consent".)