News for hackers. Stuff that matters.
Re:Show HN: News Searching API by jslakro on Tuesday July 07, 2020 @12:39PM (#23760937)
How do you get news from its sources?
Re:Show HN: News Searching API by caballeto on Tuesday July 07, 2020 @01:03PM (#23761290)
First, I extract the links to the news articles by searching them on the home page or sitemaps. Then, for each article I scrap raw html and extract key things like title, description, publish date, authors, etc. Usually each news article has a schema embedded within it, so it greatly simplifies things of extracting article details.
Re:Show HN: News Searching API by erwinkle on Thursday July 09, 2020 @02:55PM (#23783712)
Do you scrape using web requests? Or do you use headless browsing to load the page and extract the contents out of the DOM?
Re:Show HN: News Searching API by caballeto on Thursday July 09, 2020 @05:38PM (#23785527)
I use standard web requests to get raw html and parse the data from it, no headless browsing. I guess all the news pages I've met so far were static.