Re:I was wrong about robots.txt
by EPendragon on Thursday July 17, 2025 @09:41AM
(#44593308)
That was my hunch. My initial post on robots.txt: https://evgeniipendragon.com/posts/i-am-disallowing-all-craw... - revolved around blocking AI models from doing that because I do not believe that it will bring more traffic to my website - it will use the content to keep people using their service. I might be proven wrong in the future, but I do not see why they would want to let go of an extra opportunity to increase retention.
[ Parent
| on HN ]
There are no comments on this comment yet.