Re:Google’s robots.txt parser is now open source
by pedrorijo91 on Monday July 01, 2019 @03:17PM
(#20328765)
can this been seen as a initiative to make google robots.txt parser the internet standard? every webmaster will want to be compliant with google corner cases...
[ Parent
| on HN ]