Re:Hacker News on BigQuery: Now with daily updates. Top doma...
by koolba on Friday March 17, 2017 @05:59PM
(#13898337)
How does the data get to BigQuery? Anything special/fun or just repeatedly polling the API endpoint?
[ Parent
| on HN ]
Re:Hacker News on BigQuery: Now with daily updates. Top doma...
by fhoffa on Friday March 17, 2017 @06:03PM
(#13898380)
I'm not sure if we are ready to document the process, but I can tell you that having an API based on Firebase helps a lot:
Re:Hacker News on BigQuery: Now with daily updates. Top doma...
by koolba on Friday March 17, 2017 @06:24PM
(#13898516)
I'd be interested in reading up on it if you ever do write it up.
I put something similar together (for HN data). One hitch is tracking historical changes as I don't think you can get a raw feed of it so stuck with polling a historical range to ensure you get the latest version of a comment.