News for hackers. Stuff that matters.

A comment on Brandchirps – Simple Brand Monitoring. Parent comment

Re:Brandchirps – Simple Brand Monitoring by corentin88 on Sunday January 19, 2020 @03:38AM (#22090223)
Even if it’s just http(s) requests that’s a lot of data to find & crawl. The bandwidth costs are probably insane.
[ Parent | on HN ]
Re:Brandchirps – Simple Brand Monitoring by JamesGreene on Sunday January 19, 2020 @10:09AM (#22091787)
I have a background in scraping from prior projects over the last decade.

Bandwidth is not a concern for projects like this at a lot of hosting/VPS providers.

[ Parent | on HN ]
Re:Brandchirps – Simple Brand Monitoring by epoch_100 on Sunday January 19, 2020 @10:03AM (#22091762)
Data ingress is usually free, which really cuts down on costs when scraping. If you can do everything in-memory, it's surprisingly cheap. The important bit is being respectful of robots.txt files and not overloading small sites with too many requests.
[ Parent | on HN ]