News for hackers. Stuff that matters.

A comment on Google’s robots.txt parser is now open source. Parent comment

Re:Google’s robots.txt parser is now open source by LunaSea on Tuesday July 02, 2019 @05:10AM (#20333416)
But they do have the actual content since they show the meta title an description on top of what I assume is heavy NLP to drive the search engine itself.
[ Parent | on HN ]

There are no comments on this comment yet.