News for hackers. Stuff that matters.
Re:Search all Craigslist cities for a remote job by Reedx on Monday April 08, 2019 @09:44PM (#19610954)
> I found that their anti-scraping defenses were really difficult to get around

What sort of defenses are they using?

Re:Search all Craigslist cities for a remote job by smt88 on Tuesday April 09, 2019 @11:46AM (#19615830)
They had fantastic detection of headless Chrome and curl requests, a thorough IP blacklist, aggressive rate limiting, and possibly some JS stuff.
Re:Search all Craigslist cities for a remote job by stevenwoo on Tuesday April 09, 2019 @04:23AM (#19612757)
I was a contractor at a company where one of the divisions did Craiglist scraping as part of the business model and they had a closet full of laptops doing the scraping part of the job - thirdhand what I heard was separate PC's were necessary for the anti-scraping workaround they were using.