It's polite, but the teeth behind it is that it's much easier to secure a CFAA conviction for scraping if you can show evidence that you specifically forbade a scraper from accessing your site (via robots.txt, authentication, or IP blocks) and they circumvented that restriction than if you gave them no notice that you dislike what they're doing.

Legit search engines and aggregators have a strong incentive not to kill the golden goose by getting sued, particularly when the vast majority of sites actually want to be indexed.