Re:What gets to the front page of Hacker News? A data project
by dredmorbius on Thursday June 29, 2023 @01:48PM
(#36524001)
So, oddly enough, I've also been looking at HN front-page characteristics, based on the same corpus (the "past" page links). And that whole section on caveats over what that archive represents is something I could have written... The front page, both in its dynamic and archived forms is strongly subject to many influences in complex ways.
A couple of tips:
- It's possible to crawl the page using wget, given a reasonable delay. The full collection from 2007 to present (I'd done my first crawl in late May of this year) took a couple of days. Updates to that happen in seconds.
- I break down data by date, story position (e.g., rank 1--30), submitted site (if present), points (votes), comments, and submitter, as well as title.
- I'm working on classifying titles. The original question prompting my analysis was what US states get the most love from HN (NY, CA, WA*, TX, and CO are the top 5). I'd expanded that US and globally-significant cities, and been doing some tuple-based ngram analysis, though that gets pretty hairy.
For 2022 (most recent complete year), the top 40 submitted front-page sites are:
(The "mean" values are the arithmetic mean of points (votes) and comments by domain.)
For 2023, there've only been 10 TechCrunch items (through 21-6-2023), well below trend:
Ubuntu 22.04 LTS servers and phased apt updates
Twitterrific has been discontinued
DuckDB – An in-process SQL OLAP database management system
Shane Pitman, leader of the warez group Razor 1911: life after prison (2005)
Nearly 40% of software engineers will only work remotely
Htmx 1.9.0 has been released
Geometry Central: library of data structures, algorithms for geometry processing
Google Authenticator now supports Google Account synchronization
I Wrote an Activitypub Server in OCaml: Lessons Learnt, Weekends Lost
In New Paradox, Black Holes Appear to Evade Heat Death
I'll note that breaking stories down by site will tend to obscure categories, as frequently-submitted sites (say, NY Times) will crowd out many individual blogs. I could probably do some manual classification based on sites, including, say, all categories of Twitter (currently broken out by user/account), and might look into that.
One of the most surprising facts to jump out to me is how much nytimes.com has fallen since 2019. It had previously been in the top-4 submitted sites pretty consistently, and single top for 2014--2019, but fell to 7th in 2020 and 9th in 2021, recovering to 5 in 2022.
That includes a number of findings (and testing/debugging notes), including: mentions of Reddit by year, mentions of the FP-500 companies (top-10: Apple, Microsoft, Amazon, Intel, Tesla, Netflix, IBM, Adobe, Oracle, and AT&T, though Google under various terms (Google, Alphabet, YouTube, Android) nearly doubles the top-ranked Apple, and no, adding in iPhone, iPad, MacBook, etc., doesn't help), trends in votes and comments by story position (interesting IMO), overall submission success rate (a hair under 3%), mentions of the FP Top 100 Global Thinkers in titles (reprising an old study of mine of numerous online sites), a look at the Leaders characteristics, what HN cares about being down, and, well, ... things: <https://toot.cat/@dredmorbius/110454128168815763>
________________________________
Notes:
* "Washington" can of course designate both a city and a state, amongst other things, and it turns out that the string is dominated by references to the Washington Post, much as "New York" is by the New York Times. But the list gives the naive ranking. Adding in "Silicon Valley" and "San Francisco" put California well on top.
Edits: Some in situ updates as I think of things. Sorry!
Re:What gets to the front page of Hacker News? A data project
by gagejustins on Thursday June 29, 2023 @01:59PM
(#36524203)
Oh man, this is awesome. I learned a lot from collecting this data and one of the big takeaways for me was how diverse the set of news sources on HN is (to your point, very little "traditional" journalism here). Glad you're doing this!
Re:What gets to the front page of Hacker News? A data project
by dredmorbius on Thursday June 29, 2023 @02:14PM
(#36524486)
Drop me a line if you'd like to discuss this / share w/ reports. username at Protonmail.
I'm sort of a Can Haz All The Tables sort of guy, and I'm largely processing via awk (and a few other shell tools). So pasting that here would get a bit tedious...
It's also been interesting to look at how HN has, and hasn't, changed over the years. Your categorical analysis would be an interesting filter to look at over time, especially regarding accusations that HN is drifting in various directions.
The other bit that stands out to me is how constrained a set the front page is (30 slots per day, 10,950 per year, 10,980 in a leap year), as well as how thin submission titles are for gleaning meaning and context (I'm ... somewhat frustrated by this). Though there is clearly signal that gets through.
I don't have time-of-day granularity, but can look at day-of-week (and have) and month-of-year (not yet) looking for seasonality. DoW has been interesting (usually peaks Tue/Wed, starts trailing off on Fri, Sat & Sun are low points, based on votes/comments, but give higher odds of a given submission landing).
Re:What gets to the front page of Hacker News? A data project
by dredmorbius on Friday June 30, 2023 @07:56AM
(#36533318)
For those interested in the Ongoing Saga of HN Front Page Analyticcs, I've been posting occasional updates to the above site-based classification (~60% of posts now classified) to the Fediverse: <https://toot.cat/@dredmorbius/tagged/HackerNewsAnalytics>
(It's a bit much to dump massive tables to HN, I'm trying to keep that to a bearable minimum.)
Re:What gets to the front page of Hacker News? A data project
by dredmorbius on Friday June 30, 2023 @05:11PM
(#36542345)
Having played with classifying sites for much of the past day, I've assigned a classification to just under 30% of them, which classifies just under 64% of all posts.
The remaining unclassified sites average about 1.7 posts each (there are a few with as many as 20 posts), but there are minimal gains for additional classification.
I'm starting now with running an analysis over the full archive to come up with trends-by-classification over years.
The top-20 classifications (by story) are:
1 64777 36.21% UNCLASSIFIED
2 22481 12.57% blog
3 15106 8.44% general news
4 13769 7.70% tech news
5 12709 7.10% programming
6 8459 4.73% academic / science
7 8200 4.58% corporate comm.
8 7294 4.08% n/a
9 5311 2.97% business news
10 3798 2.12% general interest
11 2151 1.20% social media
12 2048 1.14% software
13 1613 0.90% technology
14 1432 0.80% video
15 1144 0.64% general information (wiki)
16 1006 0.56% government
17 724 0.40% misc documents
18 720 0.40% law
19 702 0.39% tech discussion
20 620 0.35% science news
I've got a total of 60 classifications which ... seems a bit high, and I'm looking at ways of slimming that down. It's also a bit confused, as some is classified by topic ("programming", "networking" "database", "cryptocurrency", "crowdfunding"), some by source ("corporate comm." is any post that originates from an identifiable company communicating as that company), and general format ("blog" includes 5,306 sites, and spans a wide range of topics). The distinction between, say, "tech news" and "blog" is somewhat ambiguous, and there are a few blogs which should be classified as "corporate comms.". But in all there's a rough sense of what types of content are being posted, and I'd really like to see the change over time.
Re:What gets to the front page of Hacker News? A data project
by dredmorbius on Friday June 30, 2023 @08:03AM
(#36533386)
So, some further thoughts on your methodology:
- It's comprehensive. That's ... admirable, but not necessarily efficient in data analysis. There's a lot to be said for both random sampling and inference.
- You might get more mileage by looking at the top-n stories of a given day. I'd suggest 3--5 items. There's a considerable fall-off in activity from storypos 1 to storypos 30 (1st to 30th items on the front page archive), which is one of the dimensions I've looked at.
- The thought that's occurred to me over the past few days is that this seems like a natural area in which LLM / GPT techniques might be used to classify posts given training data.
- Tuple and ngram analysis can also turn up interesting patterns. Here it's useful to have a base corpus from which universal tendencies can be inferred, and to look at statistically improbably terms which occur both from the HN subject corpus to the universal corpus (terms and phrases which HN finds significant), as well as changing trends over time within the HN corpus.
- Day-of-week and month-of-year analysis can also show interesting patterns, and I've looked at a bit of the first. I'd really like to know if there's an HN "September" (on an annual basis).
- I took a look at your data and ... spreadsheets. Maybe I'm old-school, but flatfiles and gawk are really my style.
A couple of tips:
- It's possible to crawl the page using wget, given a reasonable delay. The full collection from 2007 to present (I'd done my first crawl in late May of this year) took a couple of days. Updates to that happen in seconds.
- I break down data by date, story position (e.g., rank 1--30), submitted site (if present), points (votes), comments, and submitter, as well as title.
- I'm working on classifying titles. The original question prompting my analysis was what US states get the most love from HN (NY, CA, WA*, TX, and CO are the top 5). I'd expanded that US and globally-significant cities, and been doing some tuple-based ngram analysis, though that gets pretty hairy.
For 2022 (most recent complete year), the top 40 submitted front-page sites are:
TechCrunch, BTW, lands at #41: (The "mean" values are the arithmetic mean of points (votes) and comments by domain.)For 2023, there've only been 10 TechCrunch items (through 21-6-2023), well below trend:
I'll note that breaking stories down by site will tend to obscure categories, as frequently-submitted sites (say, NY Times) will crowd out many individual blogs. I could probably do some manual classification based on sites, including, say, all categories of Twitter (currently broken out by user/account), and might look into that.One of the most surprising facts to jump out to me is how much nytimes.com has fallen since 2019. It had previously been in the top-4 submitted sites pretty consistently, and single top for 2014--2019, but fell to 7th in 2020 and 9th in 2021, recovering to 5 in 2022.
I've also paired my own analysis with a 2022 study published by Whaly.io based on the HN API and all content submitted: <https://whaly.io/posts/hacker-news-2021-retrospective>
I've been somewhat live-bloogging my analysis on the Fediverse under the #HackerNewsAnalytics hashtag:
<https://toot.cat/@dredmorbius/tagged/HackerNewsAnalytics>
That includes a number of findings (and testing/debugging notes), including: mentions of Reddit by year, mentions of the FP-500 companies (top-10: Apple, Microsoft, Amazon, Intel, Tesla, Netflix, IBM, Adobe, Oracle, and AT&T, though Google under various terms (Google, Alphabet, YouTube, Android) nearly doubles the top-ranked Apple, and no, adding in iPhone, iPad, MacBook, etc., doesn't help), trends in votes and comments by story position (interesting IMO), overall submission success rate (a hair under 3%), mentions of the FP Top 100 Global Thinkers in titles (reprising an old study of mine of numerous online sites), a look at the Leaders characteristics, what HN cares about being down, and, well, ... things: <https://toot.cat/@dredmorbius/110454128168815763>
________________________________
Notes:
* "Washington" can of course designate both a city and a state, amongst other things, and it turns out that the string is dominated by references to the Washington Post, much as "New York" is by the New York Times. But the list gives the naive ranking. Adding in "Silicon Valley" and "San Francisco" put California well on top.
Edits: Some in situ updates as I think of things. Sorry!
I'm sort of a Can Haz All The Tables sort of guy, and I'm largely processing via awk (and a few other shell tools). So pasting that here would get a bit tedious...
It's also been interesting to look at how HN has, and hasn't, changed over the years. Your categorical analysis would be an interesting filter to look at over time, especially regarding accusations that HN is drifting in various directions.
The other bit that stands out to me is how constrained a set the front page is (30 slots per day, 10,950 per year, 10,980 in a leap year), as well as how thin submission titles are for gleaning meaning and context (I'm ... somewhat frustrated by this). Though there is clearly signal that gets through.
I don't have time-of-day granularity, but can look at day-of-week (and have) and month-of-year (not yet) looking for seasonality. DoW has been interesting (usually peaks Tue/Wed, starts trailing off on Fri, Sat & Sun are low points, based on votes/comments, but give higher odds of a given submission landing).
You might want to look at Whaly's work as well (I'd edited it into my larger top-level comment above: <https://whaly.io/posts/hacker-news-2021-retrospective>).
(It's a bit much to dump massive tables to HN, I'm trying to keep that to a bearable minimum.)
The remaining unclassified sites average about 1.7 posts each (there are a few with as many as 20 posts), but there are minimal gains for additional classification.
I'm starting now with running an analysis over the full archive to come up with trends-by-classification over years.
The top-20 classifications (by story) are:
I've got a total of 60 classifications which ... seems a bit high, and I'm looking at ways of slimming that down. It's also a bit confused, as some is classified by topic ("programming", "networking" "database", "cryptocurrency", "crowdfunding"), some by source ("corporate comm." is any post that originates from an identifiable company communicating as that company), and general format ("blog" includes 5,306 sites, and spans a wide range of topics). The distinction between, say, "tech news" and "blog" is somewhat ambiguous, and there are a few blogs which should be classified as "corporate comms.". But in all there's a rough sense of what types of content are being posted, and I'd really like to see the change over time.- It's comprehensive. That's ... admirable, but not necessarily efficient in data analysis. There's a lot to be said for both random sampling and inference.
- You might get more mileage by looking at the top-n stories of a given day. I'd suggest 3--5 items. There's a considerable fall-off in activity from storypos 1 to storypos 30 (1st to 30th items on the front page archive), which is one of the dimensions I've looked at.
- The thought that's occurred to me over the past few days is that this seems like a natural area in which LLM / GPT techniques might be used to classify posts given training data.
- Tuple and ngram analysis can also turn up interesting patterns. Here it's useful to have a base corpus from which universal tendencies can be inferred, and to look at statistically improbably terms which occur both from the HN subject corpus to the universal corpus (terms and phrases which HN finds significant), as well as changing trends over time within the HN corpus.
- Day-of-week and month-of-year analysis can also show interesting patterns, and I've looked at a bit of the first. I'd really like to know if there's an HN "September" (on an annual basis).
- I took a look at your data and ... spreadsheets. Maybe I'm old-school, but flatfiles and gawk are really my style.