News for hackers. Stuff that matters.
Re:Web Scraping in 2016 by sjwright on Tuesday August 23, 2016 @11:32PM (#12349483)
And if you have some way to identify yourself as a potential competitor to google and not some jackass trying to scrape email addresses or spam comments forms, I'm all ears.
Re:Web Scraping in 2016 by greglindahl on Tuesday August 23, 2016 @11:52PM (#12349550)
A majority of the websites that blekko, a google competitor, contacted to ask for robots.txt access ignored us.
Re:Web Scraping in 2016 by sjwright on Wednesday August 24, 2016 @12:40AM (#12349718)
I agree, it's a difficult conundrum. It sucks.
Re:Web Scraping in 2016 by greglindahl on Wednesday August 24, 2016 @01:20AM (#12349843)
There are worse barriers to entry for a search engine! DMCA take-downs... Right to be forgotten... click history...