Web crawlers generally allow sites to remove them from the index.

Are there any crawlers used for commercial purposes which refuse to remove sites from an index if they ask? The distinction from OpenAI is that there is no way to be removed from openai's training set.

You can remove yourself from the crawler not but not from what they previously crawled.

BIf a copy of the downloaded file is redistributed or used in possibly other ways that infringe on copyrights, THAT I could understand, but suddenly making the act of just downloading the file (assuming it is made legally available to the public). But if the downloaded file is analyzed by some software and then thrown away, I don't see how that is infringing copyrights, more than say downloading an image to decompress it and scale it to display on a screen (then throw it away once it is no longer needed).