Re:Wayback machine gets a facelift, new features
by greglindahl on Monday November 04, 2013 @07:28PM
(#6672213)
Speaking as an upstart search engine guy (blekko) who also has a bunch of webpages and a huge robots.txt, that's a bad idea. Such a crawler would be knocking down webservers by running expensive scripts and clicking links that do bad things like deleting records from databases or reverting edits in wikis. You don't want to go there.
[ Parent
| on HN ]
Re:Wayback machine gets a facelift, new features
by derekp7 on Monday November 04, 2013 @08:45PM
(#6672649)
Really? I was always taught that search engines only do "get" requests, and anything that modifies data is in a "post" request. Are there really that many broken web sites out there, that hasn't already fallen victim to crawlers that ignore robots.txt?
Re:Wayback machine gets a facelift, new features
by greglindahl on Monday November 04, 2013 @09:17PM
(#6672773)
Yes, there are a lot of broken websites out there.