News for hackers. Stuff that matters.

A comment on CNET is deleting old articles to try to improve its Google Search ranking. Parent comment

Re:CNET is deleting old articles to try to improve its Googl... by amatecha on Wednesday August 09, 2023 @07:48PM (#37069959)
Yeah wait, what? I hope that's incorrect. Updating robots.txt to disallow should only omit content from that point onward... It shouldn't be retroactive. What if there's a new owner of a respective domain, for example?
[ Parent | on HN ]
Re:CNET is deleting old articles to try to improve its Googl... by rany_ on Wednesday August 09, 2023 @08:04PM (#37070073)
It isn't, I really wish that instead of wiping DECADES of history; it only applies to content that was archived from the day of the domain's registration. I think this is slightly more reasonable, but I imagine they simply don't have access to such data.
[ Parent | on HN ]
Re:CNET is deleting old articles to try to improve its Googl... by pseudalopex on Wednesday August 09, 2023 @08:48PM (#37070420)
Simply requiring domain owners contact archive.org to remove old snapshots would be better than applying robots.txt retroactively.
[ Parent | on HN ]
Re:CNET is deleting old articles to try to improve its Googl... by pseudalopex on Wednesday August 09, 2023 @08:44PM (#37070386)
It was correct. And archive.org used this self inflicted problem to justify disregarding robots.txt.

https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

[ Parent | on HN ]
Re:CNET is deleting old articles to try to improve its Googl... by npunt on Wednesday August 09, 2023 @09:27PM (#37070730)
Oh thank you for the update, that's great news! Agree with their position 100%.
[ Parent | on HN ]
Re:CNET is deleting old articles to try to improve its Googl... by pseudalopex on Wednesday August 09, 2023 @10:25PM (#37071188)
> Agree with their position 100%.

Including the part where they blamed robots.txt for their decision surprising almost everyone to apply it retroactively?

[ Parent | on HN ]
Re:CNET is deleting old articles to try to improve its Googl... by npunt on Thursday August 10, 2023 @01:15AM (#37072236)
no I didn't see that in there, I just agree that robots.txt makes sense for web crawlers but not archival purposes
[ Parent | on HN ]