News for hackers. Stuff that matters.

A comment on A search engine that favors text-heavy sites and punishes modern web design. Parent comment

Re:A search engine that favors text-heavy sites and punishes... by kews on Thursday September 16, 2021 @12:33PM (#28553882)
Do you know what proportion of the texty web instructs unknown crawlers to go away (or blocks them)?
[ Parent | on HN ]
Re:A search engine that favors text-heavy sites and punishes... by marginalia_nu on Friday September 17, 2021 @07:30AM (#28564128)
It's hard to give numbers, it doesn't seem to be very many, but losing out on a few key sites does make a pretty big impact.

You see stuff like this sometimes, makes me a bit sad.

https://linux.die.net/robots.txt

[ Parent | on HN ]