Re:Web Scraping: Bypassing “403 Forbidden,” captchas, and more
by brilliantcode on Thursday March 16, 2017 @05:00PM
(#13889140)
interesting. are you seeing any product/market fit for this?
[ Parent
| on HN ]
Re:Web Scraping: Bypassing “403 Forbidden,” captchas, and more
by jakubbalada on Thursday March 16, 2017 @06:56PM
(#13889909)
We see a lot of users who needs data from the web or APIs for sites which doesn't have one. Just not all of them can code and we have to scale custom development.
Re:Web Scraping: Bypassing “403 Forbidden,” captchas, and more
by brilliantcode on Thursday March 16, 2017 @07:47PM
(#13890179)
Are these developers? Business people? I'm curious because we've been searching for a tool like this for a while but ultimately management thought it was a bad idea to rely on scraping, there's simply no replacement for a REST api.
Re:Web Scraping: Bypassing “403 Forbidden,” captchas, and more
by jakubbalada on Friday March 17, 2017 @03:52AM
(#13892300)
Both - developers on a free plan using own RSS for sites without one and business people (mainly startups) building their products on top of Apifier.
Typical use is an aggregator that needs common API for all partners who are not able to provide it. So they have running API on Apifier in an hour. It might break once in a while - than you have to update your crawler (not that often if you use internal AJAX calls).
Re:Web Scraping: Bypassing “403 Forbidden,” captchas, and more
by brilliantcode on Friday March 17, 2017 @03:54PM
(#13897406)
I see, so there's not much value beyond startups and bootstrappers.
I feel like it's a hard sell to enterprises. Scraping is viewed inferior to an API so it makes sense for enterprises to just pay the target website for access to the data.
Re:Web Scraping: Bypassing “403 Forbidden,” captchas, and more
by jakubbalada on Saturday March 18, 2017 @03:47AM
(#13900365)
It's also hard to get direct access to the data.
But you're right it's a hard sell to enterprises although we have some (e.g. real estate developer creating pricing maps)