News for hackers. Stuff that matters.
Re:Wayback machine gets a facelift, new features by jccalhoun on Monday November 04, 2013 @05:55PM (#6671791)
I'm surprised that some upstart search engine hasn't made a selling point that they ignore robots.txt and claim they search the pages google doesn't or something.
Re:Wayback machine gets a facelift, new features by greglindahl on Monday November 04, 2013 @07:28PM (#6672213)
Speaking as an upstart search engine guy (blekko) who also has a bunch of webpages and a huge robots.txt, that's a bad idea. Such a crawler would be knocking down webservers by running expensive scripts and clicking links that do bad things like deleting records from databases or reverting edits in wikis. You don't want to go there.
Re:Wayback machine gets a facelift, new features by derekp7 on Monday November 04, 2013 @08:45PM (#6672649)
Really? I was always taught that search engines only do "get" requests, and anything that modifies data is in a "post" request. Are there really that many broken web sites out there, that hasn't already fallen victim to crawlers that ignore robots.txt?
Re:Wayback machine gets a facelift, new features by greglindahl on Monday November 04, 2013 @09:17PM (#6672773)
Yes, there are a lot of broken websites out there.
Re:Wayback machine gets a facelift, new features by blueblob on Tuesday November 05, 2013 @02:33AM (#6673716)
I noticed this today. Googling "united check in" and clicking the link "check" gave me a link that told me the confirmation number that I entered was invalid though I never entered one.
Re:Wayback machine gets a facelift, new features by ISL on Monday November 04, 2013 @06:31PM (#6671953)
If their IPs became known, they might get blocked?