Show HN: Crul || Query Any Webpage or API

Re:Show HN: Crul || Query Any Webpage or API by portInit on Tuesday February 28, 2023 @01:24PM (#34972855)

At this point we're considering it a foundational concept to build around - web content changes, so our best option currently is to make the query as easy as possible to change, and alert when things break.

We have done some preliminary work in some AI or other intelligence for pattern recognition to be able to handle structural changes better, but still have lots of work.

But the expanding and querying concepts also make a lot of sense with APIs, which tend to be a little more stable.

[ Parent | on HN ]

Re:Show HN: Crul || Query Any Webpage or API by robbs on Tuesday February 28, 2023 @02:55PM (#34973960)

IMO, this is the hardest part of maintaining a web scraper. We had ~100 scripts to scrape ~1000 clients' sites and it was, at minimum, 50 hours a week to keep up with changes.

The second hardest part was 30% of our clients all used the same hosting provider, which would start to fail at 10-20 req/s. We had to throttle the sites by IP, cluster-wide.

[ Parent | on HN ]

Re:Show HN: Crul || Query Any Webpage or API by portInit on Tuesday February 28, 2023 @03:02PM (#34974053)

This makes sense and I am curious about this. Was there consistency between those 1k client sites or were they all rather different? Mind if I reach out?

[ Parent | on HN ]