News for hackers. Stuff that matters.
Re:Web Scraping in Python – The Complete Guide by thomasisaac on Tuesday February 20, 2024 @11:09AM (#39442988)
We've used ScraperAPI for a long time: https://www.scraperapi.com/

Couldn't recommend them more.

Re:Web Scraping in Python – The Complete Guide by croemer on Tuesday February 20, 2024 @11:30AM (#39443293)
I've used ScrapingBee which has similar pricing and has worked well, can't say which one is better: https://www.scrapingbee.com/
Re:Web Scraping in Python – The Complete Guide by screye on Tuesday February 20, 2024 @11:54AM (#39443608)
We used to be on ScraperAPI, but moved to ScrapingBee after more frequent failures from ScraperAPI. If your scraping needs have realtime requirements, then I'd recommend ScrapingBee.
Re:Web Scraping in Python – The Complete Guide by thomasisaac on Tuesday February 20, 2024 @12:55PM (#39444430)
Weird, we found the exact opposite - what were you scraping?

ScrapingBee really struggles on so many domains - ScraperAPI is almost as good as Brightdata when it comes to hard to beat sites.

Re:Web Scraping in Python – The Complete Guide by daolf on Tuesday February 20, 2024 @01:12PM (#39444654)
Hi Thomas, really sorry you had a bad experience with ScrapingBee.

Would you mind sending me the account you used as I wasn't able to find anything under Thomas Isaac or Tillypa and couldn't see what was going wrong then.

I'm sure your comment has nothing to do with the fact that you share the same investor as ScraperAPI but I just wanted be sure.

Re:Web Scraping in Python – The Complete Guide by DoodahMan on Tuesday February 20, 2024 @08:06PM (#39449047)
any HN discount by chance? ;) i'm testing y'all out now for a time-sensitive scrape job that must be done by Mar 1st.
Re:Web Scraping in Python – The Complete Guide by OlikEten on Wednesday February 21, 2024 @07:45AM (#39453099)
I am surprised nobody mentioned https://apify.com/ and they even offer discount for YC startups as ex-graduate from the YC Combinator program
Re:Web Scraping in Python – The Complete Guide by DoodahMan on Thursday February 22, 2024 @04:52PM (#39473801)
btw your free trial seems totally broken; have reached out via contact form.
Re:Web Scraping in Python – The Complete Guide by rustdeveloper on Tuesday February 20, 2024 @01:08PM (#39444594)
I'm using Scraping Fish because of their pay-as-you-go style pricing as opposed to subscription with monthly scraping volume commitment. And they don't charge extra credits for JS rendering or residential proxies because the cost of each request is the same: https://scrapingfish.com