Re:Show HN: I built a service to discover and monitor rapidl...
by saturday14 on Saturday April 04, 2020 @09:43AM
(#22778167)
Very cool. I guess as you grow, you could get more partners and the data will keep getting more and more accurate. I wish big guys like Amazon released aggregated sales data.
How do you get access to Amazon searches, google searches etc? do you have to pay for those?
Re:Show HN: I built a service to discover and monitor rapidl...
by mikerubini on Saturday April 04, 2020 @09:46AM
(#22778189)
Absolutely. I don't pay to access data. All of my projects are around collecting data. I'm a master scraper ️
If you are curious, at https://rubini.solutions you can check out my other SaaS products =)
Re:Show HN: I built a service to discover and monitor rapidl...
by saturday14 on Saturday April 04, 2020 @10:22AM
(#22778362)
Just checked your list of products, very cool!
I've always wondered about building a career with scraping, but never got around to it. One reason is the legality of scraping - big sites like Craigslist have gone after (and shut down) smaller projects. I believe priceonomics (YC company) built a big business with scraping.
Re:Show HN: I built a service to discover and monitor rapidl...
by riku_iki on Saturday April 04, 2020 @10:48AM
(#22778515)
And how do you scrap Google data? They don't ban you or ask to enter "I am not robot" captcha?
Re:Show HN: I built a service to discover and monitor rapidl...
by mikerubini on Saturday April 04, 2020 @12:35PM
(#22779337)
No, as I mentioned, I'm a master scraper.
Re:Show HN: I built a service to discover and monitor rapidl...
by saturday14 on Saturday April 04, 2020 @04:28PM
(#22781243)
Are you not breaking their ToS though?
Re:Show HN: I built a service to discover and monitor rapidl...
by mikerubini on Saturday April 04, 2020 @04:29PM
(#22781252)
Nope, everything is ok
Specifically, it relies on Google Trends, Google News, Google searches, Amazon searches, Twitter searches and YouTube searches (for now!)
We also have a partner for e-commerce data (usecart.com)