Re:Show HN: News Searching API
by caballeto on Tuesday July 07, 2020 @01:03PM
(#23761290)
First, I extract the links to the news articles by searching them on the home page or sitemaps. Then, for each article I scrap raw html and extract key things like title, description, publish date, authors, etc. Usually each news article has a schema embedded within it, so it greatly simplifies things of extracting article details.
[ Parent
| on HN ]
Do you scrape using web requests? Or do you use headless browsing to load the page and extract the contents out of the DOM?