News for hackers. Stuff that matters.
Re:Hacker News on BigQuery: Now with daily updates. Top doma... by koolba on Friday March 17, 2017 @05:59PM (#13898337)
How does the data get to BigQuery? Anything special/fun or just repeatedly polling the API endpoint?
Re:Hacker News on BigQuery: Now with daily updates. Top doma... by fhoffa on Friday March 17, 2017 @06:03PM (#13898380)
I'm not sure if we are ready to document the process, but I can tell you that having an API based on Firebase helps a lot:

- https://github.com/HackerNews/API

Re:Hacker News on BigQuery: Now with daily updates. Top doma... by koolba on Friday March 17, 2017 @06:24PM (#13898516)
I'd be interested in reading up on it if you ever do write it up.

I put something similar together (for HN data). One hitch is tracking historical changes as I don't think you can get a raw feed of it so stuck with polling a historical range to ensure you get the latest version of a comment.

Re:Hacker News on BigQuery: Now with daily updates. Top doma... by fhoffa on Friday March 17, 2017 @06:33PM (#13898561)
We need to test this - but I'm almost sure we have the latest version of each comment in BigQuery.