News for hackers. Stuff that matters.

A comment on Show HN: Monitor websites for change and get scraped data in a webhook.

Re:Show HN: Monitor websites for change and get scraped data... by dplgk on Wednesday October 30, 2019 @05:08PM (#21402746)
Writing a scraper is easy. So I assume the special sauce in these tools is caching, not getting banned or rated limited. Mostly could be done with Tor but still doesn't work with some sites like Craigslist. Is that right?
[ Parent | on HN ]
Re:Show HN: Monitor websites for change and get scraped data... by omneity on Wednesday October 30, 2019 @05:58PM (#21403269)
It depends on your usage. If you're parsimoniously checking for data, the website will not be hostile.

If your usage requires higher frequency against a given website, you'll indeed need a proxy (which we'll be rolling relatively soon, when there is demand for it).

[ Parent | on HN ]