Ask HN: Web-scraping – Do patterns/recipes exist for common scraping targets?

Posted by kisamoto on Friday September 11, 2020 @04:48AM
from the ask-hn dept.

I'm fairly familiar with web scraping/crawling however I was wondering if there is a company/tool that has re-usable modules for scraping common websites?

Examples could include: scraping article texts from news websites; extracting recipes from Good Food etc.

Rather than rewriting what others have - is there an existing library of these scrapers/crawlers to use 'out of the box'?

( 7 comments | 10 points | on HN )

Re:Ask HN: Web-scraping – Do patterns/recipes exist for comm... by abarrettwilsdon on Saturday September 12, 2020 @02:19PM (#24454293)

Not exactly what you're looking for, but there's a OSS Chrome Extension that allows you to record your actions in browser and transcribes them into Nightmare.js code:

https://github.com/segmentio/daydream

Probably the best you're going to get - most things worth scraping are worth money, and as such are not freely available

[ Parent | on HN ]

Re:Ask HN: Web-scraping – Do patterns/recipes exist for comm... by mendelevium on Friday September 11, 2020 @11:23AM (#24443788)

For extracting news articles: https://newspaper.readthedocs.io/en/latest/

[ Parent | on HN ]

Re:Ask HN: Web-scraping – Do patterns/recipes exist for comm... by kisamoto on Friday September 11, 2020 @11:27AM (#24443828)

While not general purpose templates, for news articles that seems exceptionally useful

[ Parent | on HN ]

Re:Ask HN: Web-scraping – Do patterns/recipes exist for comm... by atmosx on Friday September 11, 2020 @11:05AM (#24443596)

Try: https://www.scrapinghub.com/

[ Parent | on HN ]

Re:Ask HN: Web-scraping – Do patterns/recipes exist for comm... by kisamoto on Friday September 11, 2020 @11:28AM (#24443838)

I know ScrapingHub actually, has some useful tools but as far as I know they don't have a library of off-the-shelf scrapers for popular websites?

[ Parent | on HN ]

Re:Ask HN: Web-scraping – Do patterns/recipes exist for comm... by thedevindevops on Friday September 11, 2020 @06:23AM (#24441091)

What you're looking for sounds very open ended but the closest thing I can think of is the Huginn project on github?

[ Parent | on HN ]

Re:Ask HN: Web-scraping – Do patterns/recipes exist for comm... by kilroy123 on Friday September 11, 2020 @06:58AM (#24441310)

This is sorely needed IMO.

[ Parent | on HN ]