Re:Reddit's robots.txt disallows all web crawlers
by rawfael on Thursday July 04, 2024 @05:54AM
(#40873676)
Seems that they might be showing a different robots.txt to Google: https://x.com/pedrodias/status/1808799403798409216
[ Parent
| on HN ]
There are no comments on this comment yet.