Re:Show HN: News Searching API
by jslakro on Tuesday July 07, 2020 @12:39PM
(#23760937)
How do you get news from its sources?
[ Parent
| on HN ]
First, I extract the links to the news articles by searching them on the home page or sitemaps. Then, for each article I scrap raw html and extract key things like title, description, publish date, authors, etc. Usually each news article has a schema embedded within it, so it greatly simplifies things of extracting article details.
Do you scrape using web requests? Or do you use headless browsing to load the page and extract the contents out of the DOM?