Claude seems to follow robots.txt by default. Actually at my organization our theory is that this is why no one is finding our public results any more.