Google’s robots.txt parser is now open source

Re:Google’s robots.txt parser is now open source by LunaSea on Tuesday July 02, 2019 @02:42AM (#20332720)

Right, but how do you index a page you weren't supposed to crawl in the first place?

[ Parent | on HN ]

Re:Google’s robots.txt parser is now open source by mijamo on Tuesday July 02, 2019 @04:06AM (#20333140)

It's like recommending a book you haven't read, and newspapers do that every day.

Basically Google finds the link in other places -> oh that must be interesting, I'm indexing it, without even reading it. So they don't have the actual content, and just use the texts from the sites that link to it.

[ Parent | on HN ]

Re:Google’s robots.txt parser is now open source by LunaSea on Tuesday July 02, 2019 @05:10AM (#20333416)

But they do have the actual content since they show the meta title an description on top of what I assume is heavy NLP to drive the search engine itself.

[ Parent | on HN ]