Re:Google’s robots.txt parser is now open source
by penagwin on Monday July 01, 2019 @03:26PM
(#20328842)
I don't mean to be rude, but the rest of my comment explains why. If there's something about it that's confusing I'd love to clarify it.
[ Parent
| on HN ]