Re:Tagger News takes a subset of HN articles, analyzes using...
by var_explained on Sunday May 14, 2017 @07:20PM
(#14338061)
Oh, that was silly of us not to use BigQuery! I was just able to use that download a full million stories (though we still would have had the rate-limiting step of downloading the articles).
During a hackathon it can be hard to tell when to keep searching for an easy solution like that, as opposed to going with something slow you know will work- sometimes it turns out to be a dead end.
Thanks for the recommendations!
https://github.com/HackerNews/API
The firebase API is excellent. I have been using that to keep http://searchhn.com up to date in real time.
Also big query is updated every day with all comments and posts. https://bigquery.cloud.google.com/dataset/bigquery-public-da...
This is what I started with to update the Searchera (https://searchera.io) index which powers Searchhn