News for hackers. Stuff that matters.
Re:Show HN: Crul || Query Any Webpage or API by robbs on Tuesday February 28, 2023 @02:55PM (#34973960)
IMO, this is the hardest part of maintaining a web scraper. We had ~100 scripts to scrape ~1000 clients' sites and it was, at minimum, 50 hours a week to keep up with changes.

The second hardest part was 30% of our clients all used the same hosting provider, which would start to fail at 10-20 req/s. We had to throttle the sites by IP, cluster-wide.

Re:Show HN: Crul || Query Any Webpage or API by portInit on Tuesday February 28, 2023 @03:02PM (#34974053)
This makes sense and I am curious about this. Was there consistency between those 1k client sites or were they all rather different? Mind if I reach out?