News for hackers. Stuff that matters.
Re:How to scrape anything on the web and not get caught by amelius on Sunday April 22, 2018 @04:43AM (#16895302)
Could this be a solution?: run a website, and let your visitors do the crawling.
Re:How to scrape anything on the web and not get caught by nojvek on Sunday April 22, 2018 @11:39AM (#16896611)
You download some popular iOS game. But in the background it’s fetching a List of urls, crawling it and sending results back to mothership.

How would an average joe even know?

Re:How to scrape anything on the web and not get caught by tzahola on Sunday April 22, 2018 @05:12AM (#16895374)
Won’t work. Adversaries can taint your data by sending back fake results.
Re:How to scrape anything on the web and not get caught by amelius on Sunday April 22, 2018 @05:19AM (#16895391)
Could multiple downloads and a "consensus" algorithm solve this problem?
Re:How to scrape anything on the web and not get caught by tzahola on Sunday April 22, 2018 @06:27AM (#16895549)
Kinda, but it’s far from trivial as you would need some sort of tolerance when comparing sites with dynamic content.