News for hackers. Stuff that matters.

A comment on Google’s robots.txt parser is now open source. Parent comment

Re:Google’s robots.txt parser is now open source by areyousure on Monday July 01, 2019 @03:04PM (#20328638)
> You'll see the site doesn't want it indexed (it's disallowed in robots.txt) but the site still ranks well for a popular keyword).

Which site? [Edit: I have now found https://www.gcsbackpack.com/ on page 6 of the results, and this was presumably the intended site.]

[ Parent | on HN ]

There are no comments on this comment yet.