> When you make an HTTP request, you're accessing a piece of private property. The owner of that property has every right to decide not to let you do so.

So why don't they do that? If they're responding to a bot's HTTP requests with content, they are choosing to give the bot access.