Robots.txt for the NYT has a specific exclusion for an 1996 news article

Re:Robots.txt for the NYT has a specific exclusion for an 19... by twofort on Monday March 29, 2021 @03:26AM (#26618579)

Or it could also be that people are more often clicking on recent links or searching for "something something 2021". I know I usually give higher preference to links with more recent date attached as that just-contains-what-you-want site from 2010 will potentially be outdated by now. Of course it depends on subject of your search, but id imagine it's true more often than not in big scale.

[ Parent | on HN ]

Re:Robots.txt for the NYT has a specific exclusion for an 19... by mvolfik on Monday March 29, 2021 @05:05AM (#26619053)

This is the thing - for some things (e.g. searching for tech to use), you care about recency. But for something you absolutely don't. But the search engines magic seems to throw this all into one bucket and notice that people are more clicking the '2021' links. And the consequence of this is SEO optimization done by automatically updating all links to current year - I searched for something on 1st Jan and already got a bunch of '2021 comparisons'

[ Parent | on HN ]

Re:Robots.txt for the NYT has a specific exclusion for an 19... by rchaud on Monday March 29, 2021 @10:08AM (#26621793)

It's a common content marketing tactic to update post titles to include "for 2021" a few weeks ahead of the new year. It's a tweak to get it ranking, as every other content marketer is doing the same.

[ Parent | on HN ]