(I think) we're talking about pages served to logged-in users—i.e. the pages for files and folders in a given user's OneDrive account. Web crawlers wouldn't be able to reach those either way, so it (inconveniently from a fail-early perspective) wouldn't affect anything if the detection code does something stupid/crazy in the case of crawlers.