News for hackers. Stuff that matters.
Re:Most-mentioned links in Hacker News comments, 2006–2015 by minimaxir on Thursday November 12, 2020 @04:28PM (#25075318)
A reminder that BigQuery (as used in the query in this link) is the best way to play with Hacker News data; don't scrape HN data manually!

The `bigquery-public-data.hacker_news.full` table appears to be up to date with the most recent HN data as well (table last updated today).

However, I'm not 100% sure the query is correct for unilaterally getting all links, as running the query on the full dataset returns the same results as running it from 2006-2015. And I value my sanity enough to not fuss around with the regex.

Re:Most-mentioned links in Hacker News comments, 2006–2015 by bigdict on Thursday November 12, 2020 @05:32PM (#25076105)
What is the best way to download this dataset? Last time I messed with it I had to pay for a Google Cloud bucket and run through some awkward sequence of steps to eventually get a local copy.

I think I ended up following the advice here: https://stackoverflow.com/questions/18493533/how-to-download....

Re:Most-mentioned links in Hacker News comments, 2006–2015 by minimaxir on Thursday November 12, 2020 @05:49PM (#25076303)
That's essentially it (export the BQ table as a CSV to a Google Cloud Storage bucket, then download it from there), but you can do that entirely in the web UI, no CLI needed.

If you just want a subset of the data, run a query, then save the query as a table in your project and export from there.

Re:Most-mentioned links in Hacker News comments, 2006–2015 by wslh on Friday November 13, 2020 @06:39AM (#25081268)
Does it cost money to do that or it is only time consuming?
Re:Most-mentioned links in Hacker News comments, 2006–2015 by minimaxir on Friday November 13, 2020 @10:32AM (#25083273)
It’s not free: Only storage and egress, which are trivial.

Running the query in BQ is free up to 1 TB and the 16000 row download is free so I recommend that if necessary.

Re:Most-mentioned links in Hacker News comments, 2006–2015 by wslh on Saturday November 14, 2020 @09:08AM (#25092362)
I think an alternative option could be to have a torrent (or other file sharing mechanism) with the public HN information. Am I missing something? 16k rows seems very tiny for doing an analysis of HN.
Re:Most-mentioned links in Hacker News comments, 2006–2015 by minimaxir on Saturday November 14, 2020 @01:26PM (#25094342)
The vast majority of potential HN analysis can be done in SQL alone within BigQuery, which is the proper way to handle a data warehouse regardless.

You mostly need the raw data for ML model training which is a very niche use case.

Re:Most-mentioned links in Hacker News comments, 2006–2015 by ZeroCool2u on Thursday November 12, 2020 @09:13PM (#25078047)
You can read it directly into a pandas DataFrame. [1]

https://pandas.pydata.org/pandas-docs/stable/reference/api/p...

Re:Most-mentioned links in Hacker News comments, 2006–2015 by minimaxir on Friday November 13, 2020 @01:36AM (#25079622)
This pandas trick is for small result datasets, in which case the 16,000 row limit from BigQuery more than satisfies it.

I would not recommend using it for the full many-million-row dataset.

Re:Most-mentioned links in Hacker News comments, 2006–2015 by bigdict on Thursday November 12, 2020 @09:45PM (#25078224)
Oh no way, thank you!
Re:Most-mentioned links in Hacker News comments, 2006–2015 by simonsarris on Thursday November 12, 2020 @04:07PM (#25075068)
Funny that searchyc.com was so necessary for so long, coming in at #54 and #68 (it seems it should be higher as these should be combined). Now it just redirects to a spam/ad website, but before HN had a search bar it was very useful.

Also interesting that it contains at #55: https://news.ycombinator.com/best

But not the ostensibly more useful: https://news.ycombinator.com/active

The url "u.ly/73I" #63 is very interesting, its not seen almost anywhere else on the web (at least on Google), and is apparently spam, now, and when you click on mentions for that matter you get:

> We found no comments matching u.ly/73I

What's the deal with that one? Was it spam comments that all got removed? It may be since some of the others were spam, like this one: https://goo.gl/l5v0b

It's impressive how little spam spam (as opposed to submarines, this is where I link to PGs essay) is on HN.

Re:Most-mentioned links in Hacker News comments, 2006–2015 by duckerude on Thursday November 12, 2020 @05:21PM (#25075975)
The u.ly link seems to have been spam for shoes. The wayback machine captured it: https://web.archive.org/web/20110319115346/http://u.ly/73I
Re:Most-mentioned links in Hacker News comments, 2006–2015 by pgt on Thursday November 12, 2020 @05:53PM (#25076343)
HN has a search bar? scrolls down Oh, wow! I always use hn.algolia.com.
Re:Most-mentioned links in Hacker News comments, 2006–2015 by flobosg on Thursday November 12, 2020 @06:49PM (#25076952)
That bar uses hn.algolia.com for search as well.
Re:Most-mentioned links in Hacker News comments, 2006–2015 by vlmutolo on Thursday November 12, 2020 @06:47PM (#25076931)
I’m pretty sure the search bar just redirects to that.
Re:Most-mentioned links in Hacker News comments, 2006–2015 by wombatmobile on Thursday November 12, 2020 @09:18PM (#25078072)
The HN guidelines deserve to be #1, not just in this list, but for the whole internet.

If conversation was as civil elsewhere as it is on HN, Americans might rediscover the value of community which they have lost to mindless bickering encouraged by commercial algorithms elsewhere.

Re:Most-mentioned links in Hacker News comments, 2006–2015 by ChrisMarshallNY on Friday November 13, 2020 @04:36AM (#25080656)
I very much appreciate the urbane nature of HN, and strive to contribute in a way that brings light; not darkness.

My digital community experience harkens from the USENET days, which made the worst pissing matches on Faecesbook look like polite disagreements between scholars.

A day or two ago, someone made a real mild slap at me (I can come across as a bit tiresome, if you haven’t noticed. Take my word for it; it’s preferable to my USENET persona), using a very tired old troll technique, and someone else flagged it.

I was actually surprised it was flagged, but it does show that people take civility seriously, hereabouts.

Re:Most-mentioned links in Hacker News comments, 2006–2015 by surround on Friday November 13, 2020 @12:17AM (#25079150)
Since we’re on the topic of civil conversation, I’m going to be a bit picky:

“American” is a generalization. Accusing them is not productive to discussion.

Re:Most-mentioned links in Hacker News comments, 2006–2015 by wombatmobile on Friday November 13, 2020 @03:21AM (#25080225)
I was thinking of American social media companies with exploitative engagement algorithms that foment adversarial discourse. You are right if you are saying the users are all over the world, not just America.

Is that what you mean? Or is the term “American” problematic in some other way?

Re:Most-mentioned links in Hacker News comments, 2006–2015 by surround on Friday November 13, 2020 @09:20AM (#25082370)
I though you were referring to American users. Accusing American companies is ok.
Re:Most-mentioned links in Hacker News comments, 2006–2015 by dgritsko on Thursday November 12, 2020 @04:25PM (#25075281)
Re:Most-mentioned links in Hacker News comments, 2006–2015 by sleavey on Thursday November 12, 2020 @04:45PM (#25075512)
I'm quite surprised #3 has only been mentioned 197 times (well, a few more after this thread) in 14 years.
Re:Most-mentioned links in Hacker News comments, 2006–2015 by JacobAldridge on Thursday November 12, 2020 @06:53PM (#25076979)
Linked 197 times, possibly referenced more than that; plus it’s a neatly exaggerated example of a joke that I’d imagine has been made many more times again:

“I have a problem I’m trying to fix with X.”

“Ah, so now you have 2 problems.”

Re:Most-mentioned links in Hacker News comments, 2006–2015 by carapace on Friday November 13, 2020 @11:28AM (#25083917)
I find it very hopeful that that particular one has been linked to so many times, maybe it's sinking in? :)
Re:Most-mentioned links in Hacker News comments, 2006–2015 by snowwrestler on Thursday November 12, 2020 @11:18PM (#25078819)
Surprised that 552 is not among these. One of my favorites:

https://xkcd.com/552/

Re:Most-mentioned links in Hacker News comments, 2006–2015 by progre on Friday November 13, 2020 @04:24AM (#25080603)
My personal experience is that posting this leads to and instant surge of downvotes
Re:Most-mentioned links in Hacker News comments, 2006–2015 by KineticLensman on Friday November 13, 2020 @06:59AM (#25081355)
The semi-inevitable downvotes may be because xkcd links are seen as cheap canned injects that although relevant and often funny don't really add anything novel to a debate. And perhaps - the horror! - they might not actually always be correct in a specific case.

Disclaimer: I really love xkcd.

Re:Most-mentioned links in Hacker News comments, 2006–2015 by goto11 on Friday November 13, 2020 @09:29AM (#25082481)
Probably because someone posts "correlation does not imply causation" as a rote "I am very smart" reply to any published study, regardless of the actual contents and claims of the study.
Re:Most-mentioned links in Hacker News comments, 2006–2015 by deeg on Thursday November 12, 2020 @04:35PM (#25075411)
#11 literally changed my life. I used to debate people online and get irrationally upset when I couldn't change their opinions. Reading that xkcd was like getting a dope slap. I still debate but I rarely let myself get upset and if I do I try to hold that in my mind.
Re:Most-mentioned links in Hacker News comments, 2006–2015 by sq_ on Thursday November 12, 2020 @06:43PM (#25076898)
I'm honestly surprised that the Wisdom of the Ancients xkcd [0] isn't on the list...

[0] https://xkcd.com/979/

Re:Most-mentioned links in Hacker News comments, 2006–2015 by Exmoor on Thursday November 12, 2020 @09:59PM (#25078304)
The only thing worse than that is when they come back and say, "Oh nevermind, I figured it out." and don't post the fix.
Re:Most-mentioned links in Hacker News comments, 2006–2015 by mitchbob on Thursday November 12, 2020 @11:24PM (#25078858)
Surprised 2347, Dependency [1], didn't make the list. But it's pretty recent.

[1] https://xkcd.com/2347/

Re:Most-mentioned links in Hacker News comments, 2006–2015 by Impossible on Thursday November 12, 2020 @07:18PM (#25077165)
#3 makes sense (I believe I've posted it in a few comments) because HN has a lot of news on new standards
Re:Most-mentioned links in Hacker News comments, 2006–2015 by peterburkimsher on Thursday November 12, 2020 @08:32PM (#25077787)
Some XKCD comics have been translated to Chinese:

https://github.com/stevenliuyi/xkcd-cn/tree/master/pics

I've combined them side-by-side with the English:

https://mega.nz/folder/modDRIYB#uxRKw78Fcv6mpclovOTRbw

I'd also like to transcribe them for use with Pingtype, but didn't get around to that yet.

Re:Most-mentioned links in Hacker News comments, 2006–2015 by simonebrunozzi on Thursday November 12, 2020 @07:02PM (#25077041)
936 is my personal favorite.
Re:Most-mentioned links in Hacker News comments, 2006–2015 by Shared404 on Thursday November 12, 2020 @11:10PM (#25078774)
My understanding is that diceware is no longer considered secure though.

Aren't there wordlists that just consist of combining dictionary words, rendering diceware more dangerous than a password manager?

Re:Most-mentioned links in Hacker News comments, 2006–2015 by shakna on Friday November 13, 2020 @04:01AM (#25080475)
There have always been dictionary attacks.

You're judging it from the wrong position. Diceware is an improvement upon the usual kinds of passwords people feel forced to come up with, and then re-use everywhere.

A password manager is better than diceware, and whilst it feels like less friction to people who use them, people can be unwilling to use one.

Just rejecting passwords found in HIBP's stolen passwords list had me receive death threats and long rants about how password security isn't my responsibility and I was putting up too much friction.

Re:Most-mentioned links in Hacker News comments, 2006–2015 by Shared404 on Friday November 13, 2020 @09:03AM (#25082193)
Ah, that makes sense. Thanks for clarifying for my slow self.

Edit: Also

> rejecting passwords found in HIBP's stolen passwords list had me receive death threats and long rants about how password security isn't my responsibility

That is amazing in a horrifying sort of way. I'm sorry you went through that.

Re:Most-mentioned links in Hacker News comments, 2006–2015 by surround on Friday November 13, 2020 @01:52PM (#25085944)
The comic calculates entropy with the assumption that the attacker knows that your password is four common words. Even if they run a dictionary attack on your password, it will still be secure.

Diceware passwords are still recommended by security experts over any other method.

https://www.eff.org/dice

Re:Most-mentioned links in Hacker News comments, 2006–2015 by naveen99 on Friday November 13, 2020 @09:41AM (#25082623)
#78 -xkcd 810 broke my brain. Nice !

Will add to my list of things I’d look into if I had more time.

Re:Most-mentioned links in Hacker News comments, 2006–2015 by ellis0n on Thursday November 12, 2020 @06:08PM (#25076520)
Haha, just #927, nothing else. #927 XKCD receipt about everything
Re:Most-mentioned links in Hacker News comments, 2006–2015 by SippinLean on Thursday November 12, 2020 @08:03PM (#25077561)
I thought we'd see #91 (1053/"Ten Thousand") but it's lame having a 4-digit XKCD in the top 100
Re:Most-mentioned links in Hacker News comments, 2006–2015 by noobermin on Friday November 13, 2020 @02:28AM (#25079911)
"The submarine" was nice for its time. The reality is today astroturfing is very much a real thing especially as print media has died and the same types of firms find ways to hawk their wares on the internet.

pg also had the right idea but you could, if you desire sit further back and ask about how someone's personal self-interests affect what they write (or nowadays, post or whatever) about and how that can affect the sincerity of what they are writing.

Re:Most-mentioned links in Hacker News comments, 2006–2015 by joshu on Friday November 13, 2020 @04:44AM (#25080700)
the submarine article was originally about me (but was wrong, we hadn’t hired PR)
Re:Most-mentioned links in Hacker News comments, 2006–2015 by oblio on Friday November 13, 2020 @07:13AM (#25081417)
Well, maybe the target was wrong but the idea was still quite good.
Re:Most-mentioned links in Hacker News comments, 2006–2015 by rmorey on Thursday November 12, 2020 @09:41PM (#25078201)
browsing some of these, i'm reminded of "Cool URIs don't change": https://www.w3.org/Provider/Style/URI
Re:Most-mentioned links in Hacker News comments, 2006–2015 by saagarjha on Thursday November 12, 2020 @04:27PM (#25075313)
If I ever needed to summarize Hacker News into a single list, this would absolutely be it.
Re:Most-mentioned links in Hacker News comments, 2006–2015 by pbiggar on Thursday November 12, 2020 @09:07PM (#25078016)
Was kinda surprised to see https://circleci.com at #37. When I clicked through, turns out it was me repeatedly reminding anyone who'd listen that we existed.
Re:Most-mentioned links in Hacker News comments, 2006–2015 by zuhayeer on Thursday November 12, 2020 @05:07PM (#25075781)
Is it possible to do this for 2015 until now as well?
Re:Most-mentioned links in Hacker News comments, 2006–2015 by macieklaskus on Thursday November 12, 2020 @04:25PM (#25075283)
Very cool! It's already useful, but you could make it even more so by enabling to sort by smaller time ranges (e.g. 1 year).

It would also be interesting to see a version of this list weighted by karma scores of users who posted the links.

Edit: even better, use your own h-index ranking https://github.com/antontarasenko/smq/blob/master/reports/ha...

Re:Most-mentioned links in Hacker News comments, 2006–2015 by mattmanser on Thursday November 12, 2020 @07:23PM (#25077207)
Looks like someone made this 5 years ago from a data dump, no updates to the repo in 5 years.
Re:Most-mentioned links in Hacker News comments, 2006–2015 by anonytrary on Thursday November 12, 2020 @05:58PM (#25076404)
Nice idea! Note the scraper seems to have produced a duplicate entry:

  1. en.wikipedia.org/wiki/Betteridges_Law_of_Headlines
  2. en.wikipedia.org/wiki/Betteridge%2527s_law_of_headlines
The second is a bad link, so I am curious how that link got shared so much.
Re:Most-mentioned links in Hacker News comments, 2006–2015 by krussell on Thursday November 12, 2020 @07:07PM (#25077079)
Interesting, looks like the wikipedia title spells it Betteridge's with an apostrophe. %27 is the url encoding for the apostrophe and %25 is the url encoding for the %. So it looks like the link somehow got url encoded twice before it was shared resulting in the apostrophe becoming %27 and then the %27 becoming %2527 and producing a bad link.

Some common method of people finding and sharing that url must have had an issue where it incorrectly encoded the url twice. I wonder what it was :) .

Or perhaps the scraper has a bug where it is double url encoding the en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines link?

Re:Most-mentioned links in Hacker News comments, 2006–2015 by identity0 on Thursday November 12, 2020 @10:47PM (#25078612)
I watched https://www.youtube.com/watch?v=dBnniua6-oM a few days ago and was surprised to see is the #1 youtube link. It does have 11M views, however, and it is a great talk.
Re:Most-mentioned links in Hacker News comments, 2006–2015 by didibus on Friday November 13, 2020 @04:43AM (#25080697)
Had missed this Paul Graham one: http://www.paulgraham.com/disagree.html but it's really good.
Re:Most-mentioned links in Hacker News comments, 2006–2015 by dang on Thursday November 12, 2020 @07:39PM (#25077347)
Re:Most-mentioned links in Hacker News comments, 2006–2015 by d13 on Saturday November 14, 2020 @01:17AM (#25090645)
Can someone explain why Paul Graham? I read a few of those links and don’t get it.
Re:Most-mentioned links in Hacker News comments, 2006–2015 by luigi23 on Saturday November 14, 2020 @03:15AM (#25090983)
He is the founder of Y Combinator.
Re:Most-mentioned links in Hacker News comments, 2006–2015 by jchw on Thursday November 12, 2020 @08:16PM (#25077654)
I’m so happy to see XKCD 927, because it was my immediate first thought. It is one of the few XKCD comics I know by number, and while witty I do feel it gets misused in thought-terminating ways sometimes. Still, I feel validated in my knee-jerk reaction that it would be the top XKCD comic on the list.
Re:Most-mentioned links in Hacker News comments, 2006–2015 by kilroy123 on Friday November 13, 2020 @05:21AM (#25080884)
What about 2015 - now?
Re:Most-mentioned links in Hacker News comments, 2006–2015 by kalendos on Friday November 13, 2020 @06:09AM (#25081122)
Very cool! Inspired me to get the most upvoted XKCD comics on Reddit for 2019.

https://gist.github.com/davidgasquez/3aeaac54c5a61216ffc8f7d...

Re:Most-mentioned links in Hacker News comments, 2006–2015 by djrockstar1 on Friday November 13, 2020 @06:11AM (#25081134)
Impressive that the 17th most upvoted XKCD comic isn't even an XKCD comic!
Re:Most-mentioned links in Hacker News comments, 2006–2015 by kalendos on Friday November 13, 2020 @06:30AM (#25081224)
Nice catch! I used the first regex I could find on SO.
Re:Most-mentioned links in Hacker News comments, 2006–2015 by BenGosub on Thursday November 12, 2020 @08:44PM (#25077871)
This is gold!
Re:Most-mentioned links in Hacker News comments, 2006–2015 by abnry on Thursday November 12, 2020 @07:40PM (#25077359)
It amuses me that the Dunning–Kruger effect shows up here. It has become such a cliche for people to reference it.
Re:Most-mentioned links in Hacker News comments, 2006–2015 by ben509 on Thursday November 12, 2020 @11:03PM (#25078715)
I link this[1] every time I see it mentioned, because the standard depiction of it bears no resemblance to what's in their paper.

[1]: https://www.talyarkoni.org/blog/2010/07/07/what-the-dunning-...

Re:Most-mentioned links in Hacker News comments, 2006–2015 by Shared404 on Thursday November 12, 2020 @11:13PM (#25078789)
I'm one of today's 10,000.

Thanks!

Re:Most-mentioned links in Hacker News comments, 2006–2015 by username90 on Friday November 13, 2020 @02:58AM (#25080088)
It seems like most people greatly overestimates their understanding of the Dunning-Kruger effect.