Re:Show HN : I have developed a robots.txt full text search ...
by Xeroday on Friday August 02, 2013 @07:18PM
(#6150262)
Very cool project. I'm curious as to how you're getting all these robots - are you scraping them yourself?
[ Parent
| on HN ]
Re:Show HN : I have developed a robots.txt full text search ...
by Joyfield on Friday August 02, 2013 @07:28PM
(#6150302)
Downloading them as we speak. I have a big list of hosts/domains i have collected through spidering for my DNSDigger.com.
This is a hobby project that has grown a bit over my head hehe.
And there is no scraping needed. Robots.txt is just simple textfiles. Download and parse, repeat a couple of million times and build an index :)