Re:Search all Craigslist cities for a remote job
by smt88 on Tuesday April 09, 2019 @11:46AM
(#19615830)
They had fantastic detection of headless Chrome and curl requests, a thorough IP blacklist, aggressive rate limiting, and possibly some JS stuff.
Re:Search all Craigslist cities for a remote job
by stevenwoo on Tuesday April 09, 2019 @04:23AM
(#19612757)
I was a contractor at a company where one of the divisions did Craiglist scraping as part of the business model and they had a closet full of laptops doing the scraping part of the job - thirdhand what I heard was separate PC's were necessary for the anti-scraping workaround they were using.
What sort of defenses are they using?