Re:Show HN: News Searching API
by caballeto on Tuesday July 07, 2020 @07:26AM
(#23758030)
This is the side-project, that I have been building recently. The API allows searching over the data from many news sites and news aggregators, it also allows to easily extract whole news sites. I built it for two reasons. First, I wanted to have a source of recent news data to play with, essentially to run data analysis and train ml models on. Secondly, because alternative services had wildly high pricing, like 400-500$ per month, and I thought I could build and deploy such a system for a much cheaper price and improve my programming skills along the way. I would very much appreciate any comments/suggestions about the serive or how to make it better :)
[ Parent
| on HN ]
How are you licensing the content?
He's not.
Right, I will definitely work on adding data policy to terms of service. I think it will go along the lines, that you cannot remove the attributions/trademarks, if you are republishing the data, but doing analytics on it is ok.
It's a pretty fine line you're treading here. A lot of the reason the competition is so expensive is that they are licensing the data in bulk. There's lots of litigation in this area, see the Meltwater cases in US and UK.
If you have say 10k news sources from tens of countries around the globe, I doubt that it would be feasible to contract with every specific site out there, especially if you get data say from Google News.
Re:Show HN: News Searching API
by todsacerdoti on Thursday July 09, 2020 @11:57PM
(#23788004)
I would love to integrate your product into Pipedream.com so developers can quickly integrate Datanews with other apps such as Slack, Discord, Reddit, Google Sheets, etc. Please reach out -- tod at pipedream.
Great idea, I like it.
Nice work. Will share feedback soon after testing.
Great, thanks, would appreciate it very much. I am in process of thinking about new features, so if you have some special use cases, I could think about that.
Seemed pretty straightforward after a few tests. The only thing I noticed was regarding unique identifiers, the URL in the sources endpoint didn't seem to exactly match the URL attached to a headline? I could be wrong though, and not a big issue. Was more thinking about data cleanup and linking data easily (article/sources/authors etc).
Thanks very much for the feedback. Yes, I think your point is right. For some domains, I try to match the url with sources from "sources" list, and for others I extract source field directly from the url, I think there is a small bug when doing the former, thanks for pointing out. I will fix it soon.