Imagine you have to write a computer program to visit every page of every website on the entire Internet.

Sometimes you are going to make a mistake, and someone on the Internet is going to notice.

Except this has been going on for months, has happened a number of times, and occurs in violation of standards, like robots.txt, that were created to prevent it from happening. I'd expect it from an individual or startup, but not a large company that's been doing spidering for a long time.