How can you be so sure? Processors love locality, so they fetch the data around the requested address. Intel even used to give names to that.

So, similarly, LLM companies can see this as a signal to crawl to whole site to add to their training sets and learn from it, if the same URL is hit for a couple of times in a relatively short time period.