namiwang

Ask HN: Anyone want my realtime-snapshots dataset of Hacker News?

so I've been maintaining a project [Track HN](https://track-hacker-news.com) since 2023. it continuously takes snapshots of Hacker News, recording how the score/rank/comments of stories/comments change over time. I think this is a unique HN dataset in its own way.

I don't have enough time to maintain this project anymore and wondering if anyone would find the data useful and want to take it over and keep it alive.

## the dataset

the dataset is over 200M rows in a postgres instance, plus a clickhouse instance to speedup the analytics.

- stories/comments/users

- ~6.38M Stories, ~43.2M comments, ~1.15M users, since the epoch (2006-10-09)

- story score/comment count history - ~21.5M snapshots for ~900K stories, since 2024-03-22

- story rank history - ~145M snapshots for ~178,120 stories, since 2024-03-28

- http status of popular posts

## the product and stack

I've build a product around this dataset with a stack:

- a web app (with rails) to crawl and serve the data;

- dashboards with metabase;

- another set of dashboards with retool (retool kindly sponsored a postgres instance back then);

- a browser extension;

- few data reports;

## ideas

things you could look at from the data, for example:

- analyze activities of specific HN user

- see how stories moving through the front page

- find patterns between early score and eventual score

- see how long stories survive at different ranks

- find posting time/weekday effects, domain/user patterns, etc.

- detect/extract topics/keywords from comments for brands/products/languages, etc.

Let me know if you're interested, send your proposal/questions/cool ideas to me#nami.land. thanks!

nice job! I once created [Track HN](https://track-hacker-news.com) which does similar things! I've already archived historical data of scores/comments/ranks. It seems our projects overlap - maybe we can combine our efforts to hack something even cooler :).

Totally get the struggle — I spent way too many hours figuring which design system should I use for my next project. Ended up creating a site to compare the options: [https://rails.style]. Might help others in the same spot.