News for hackers. Stuff that matters.

What Are the Latest Scraping APIs or Services/Websites?

I'm working on a large web scraping project and looking to avoid building custom scrapers for every site using something like BeautifulSoup.

Anyone have recommendations for up-to-date scraping APIs or platforms that work well with a variety of websites?

( 7 comments | 4 points | on HN )
Re:What Are the Latest Scraping APIs or Services/Websites? by mike97 on Wednesday December 04, 2024 @07:27PM (#42323548)
I am actually working on a scraping API for serps, currently it only supports Google, may add more search engines if it gets any traction, it isn't ready yet, but hopefully it will be in a couple of days, if you want to give it a go let me know and I can give you a subscription in exchange for some feedback.
Re:What Are the Latest Scraping APIs or Services/Websites? by gbertb on Wednesday December 04, 2024 @06:16PM (#42323103)
Re:What Are the Latest Scraping APIs or Services/Websites? by jdcampolargo on Wednesday December 04, 2024 @06:21PM (#42323145)
thanks!
Re:What Are the Latest Scraping APIs or Services/Websites? by ipaddr on Wednesday December 04, 2024 @10:31PM (#42324635)
They are powered by:

https://github.com/spider-rs/spider

For those who want to self host.

Re:What Are the Latest Scraping APIs or Services/Websites? by nreece on Thursday December 05, 2024 @04:27AM (#42326468)
* Shameless plug * Our feed builder combines a web scraper, AI-powered content auto-detection and data transformer to generate custom JSON and RSS feeds for any public webpage.

https://newsloth.com

Re:What Are the Latest Scraping APIs or Services/Websites? by davidsojevic on Thursday December 05, 2024 @07:38AM (#42327635)
If you're looking to scrape any of the major search engines, eBay, Yelp, and a few others, then SerpApi has a massive number of up-to-date APIs to choose from:

https://serpapi.com/search-api

Disclaimer: I work there!

Re:What Are the Latest Scraping APIs or Services/Websites? by bootstrpppin on Thursday December 05, 2024 @11:39AM (#42329754)
We use Bright Data - so far we've found it to be pretty great, and very very cheap.

Would highly recommend