News for hackers. Stuff that matters.
Re:Web Scraping with Python by datalopers on Sunday May 15, 2022 @12:32PM (#31388658)
Headless scraping quickly becomes a very expensive approach when you try and scale the effort. I only employ it when absolutely necessary. And it’s most definitely distinguishable by any modern (incapsula, perimeterx, cloudflare) WAF.
Re:Web Scraping with Python by apienx on Sunday May 15, 2022 @01:50PM (#31389401)
The Apify library tries to address most of these issues (I'd say quite successfully). Apify.com provides a platform you can deploy the scrapers on. And yes, at scale.

There's also a software marketplace where you can order custom scrapers. 98% of the projects ran thru it have a 5-star rating (Disclaimer: I moderate that marketplace). Pro-tip: submit your project with a Gmail address to skip sales and reach me directly.

Re:Web Scraping with Python by pocket_cheese on Sunday May 15, 2022 @06:34PM (#31391813)
A federated marketplace for scrapers is an idea I have thought considerably about. If you have time, I would love to chat to a) discuss being a paying user and b) to talk about the industry and see how we could submit some scrapers.

Let me know how I can reach out to you!