News for hackers. Stuff that matters.

A comment on CNET is deleting old articles to try to improve its Google Search ranking. Parent comment

Re:CNET is deleting old articles to try to improve its Googl... by pseudalopex on Wednesday August 09, 2023 @08:44PM (#37070386)
It was correct. And archive.org used this self inflicted problem to justify disregarding robots.txt.

https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

[ Parent | on HN ]
Re:CNET is deleting old articles to try to improve its Googl... by npunt on Wednesday August 09, 2023 @09:27PM (#37070730)
Oh thank you for the update, that's great news! Agree with their position 100%.
[ Parent | on HN ]
Re:CNET is deleting old articles to try to improve its Googl... by pseudalopex on Wednesday August 09, 2023 @10:25PM (#37071188)
> Agree with their position 100%.

Including the part where they blamed robots.txt for their decision surprising almost everyone to apply it retroactively?

[ Parent | on HN ]
Re:CNET is deleting old articles to try to improve its Googl... by npunt on Thursday August 10, 2023 @01:15AM (#37072236)
no I didn't see that in there, I just agree that robots.txt makes sense for web crawlers but not archival purposes
[ Parent | on HN ]