News for hackers. Stuff that matters.
Re:Brandchirps – Simple Brand Monitoring by gorbog on Sunday January 19, 2020 @07:23AM (#22091035)
How can they crawl the entire web ( or a big portion of it) everyday without spending tens of thousands of dollars in server / bandwidth costs?

I know there are lots of open source crawlers like Stormcrawler and all, but the cost of running it at web scale everyday is prohibitive isn’t it?

Re:Brandchirps – Simple Brand Monitoring by onlyrealcuzzo on Sunday January 19, 2020 @11:55AM (#22092383)
I built something a prototype of this a year ago. But I ultimately decided that I couldn't ever see myself making close to as much money from it as I already do. I won't be sad if this service proves that wrong.

I just told customers to sign up for Google Alerts for the general web, and then I scraped new content from a bunch of sites like Reddit, Pinterest, Twitter, Instagram, Product Hunt, Hacker News.

The idea was to order everything by how many people might be seeing it and how many people were interacting with it (dis/liking).

Since most things posted on social media basically go unnoticed, I crawled things in a way that you would only see stuff any reasonable business would care about. Because of that, even for a website like Reddit, I could get all the content for an hour in only a couple thousand hits -- bandwidth wasn't anything crazy.

As a business, this gave you the ability to prioritize what you should pay attention to and where -- without giving you a bunch of noise.

Re:Brandchirps – Simple Brand Monitoring by JamesGreene on Sunday January 19, 2020 @08:35PM (#22095516)
With the way web hosting servers are set up, and specific companies that sell this hosting, bandwidth for crawling/scraping is basically $0 at most popular companies.
Re:Brandchirps – Simple Brand Monitoring by melenaos on Sunday January 19, 2020 @07:40AM (#22091119)
I suppose they query Google.
Re:Brandchirps – Simple Brand Monitoring by ikeboy on Sunday January 19, 2020 @08:20AM (#22091297)
If that's the case just use Google alerts.