News for hackers. Stuff that matters.
Re:Most-mentioned links in Hacker News comments, 2006–2015 by minimaxir on Thursday November 12, 2020 @04:28PM (#25075318)
A reminder that BigQuery (as used in the query in this link) is the best way to play with Hacker News data; don't scrape HN data manually!

The `bigquery-public-data.hacker_news.full` table appears to be up to date with the most recent HN data as well (table last updated today).

However, I'm not 100% sure the query is correct for unilaterally getting all links, as running the query on the full dataset returns the same results as running it from 2006-2015. And I value my sanity enough to not fuss around with the regex.

Re:Most-mentioned links in Hacker News comments, 2006–2015 by bigdict on Thursday November 12, 2020 @05:32PM (#25076105)
What is the best way to download this dataset? Last time I messed with it I had to pay for a Google Cloud bucket and run through some awkward sequence of steps to eventually get a local copy.

I think I ended up following the advice here: https://stackoverflow.com/questions/18493533/how-to-download....

Re:Most-mentioned links in Hacker News comments, 2006–2015 by minimaxir on Thursday November 12, 2020 @05:49PM (#25076303)
That's essentially it (export the BQ table as a CSV to a Google Cloud Storage bucket, then download it from there), but you can do that entirely in the web UI, no CLI needed.

If you just want a subset of the data, run a query, then save the query as a table in your project and export from there.

Re:Most-mentioned links in Hacker News comments, 2006–2015 by wslh on Friday November 13, 2020 @06:39AM (#25081268)
Does it cost money to do that or it is only time consuming?
Re:Most-mentioned links in Hacker News comments, 2006–2015 by minimaxir on Friday November 13, 2020 @10:32AM (#25083273)
It’s not free: Only storage and egress, which are trivial.

Running the query in BQ is free up to 1 TB and the 16000 row download is free so I recommend that if necessary.

Re:Most-mentioned links in Hacker News comments, 2006–2015 by wslh on Saturday November 14, 2020 @09:08AM (#25092362)
I think an alternative option could be to have a torrent (or other file sharing mechanism) with the public HN information. Am I missing something? 16k rows seems very tiny for doing an analysis of HN.
Re:Most-mentioned links in Hacker News comments, 2006–2015 by minimaxir on Saturday November 14, 2020 @01:26PM (#25094342)
The vast majority of potential HN analysis can be done in SQL alone within BigQuery, which is the proper way to handle a data warehouse regardless.

You mostly need the raw data for ML model training which is a very niche use case.

Re:Most-mentioned links in Hacker News comments, 2006–2015 by ZeroCool2u on Thursday November 12, 2020 @09:13PM (#25078047)
You can read it directly into a pandas DataFrame. [1]

https://pandas.pydata.org/pandas-docs/stable/reference/api/p...

Re:Most-mentioned links in Hacker News comments, 2006–2015 by minimaxir on Friday November 13, 2020 @01:36AM (#25079622)
This pandas trick is for small result datasets, in which case the 16,000 row limit from BigQuery more than satisfies it.

I would not recommend using it for the full many-million-row dataset.

Re:Most-mentioned links in Hacker News comments, 2006–2015 by bigdict on Thursday November 12, 2020 @09:45PM (#25078224)
Oh no way, thank you!