Re:28M Hacker News comments as vector embedding search dataset
by doctorslimm on Friday November 28, 2025 @03:46PM
(#46082626)
why is this not on huggingface as a dataset yet? is anyone poutine this on hugginggface?
[ Parent
| on HN ]
Re:28M Hacker News comments as vector embedding search dataset
by dylan604 on Friday November 28, 2025 @08:50PM
(#46084600)
Maybe you skimmed past this from TFA:
"Well, the first problem I had, in order to do something like that, was to find an archive with Hacker News comments. Luckily there was one with apparently everything posted on HN from the start to 2023, for a huge 10GB of total data. You can find it here: https://huggingface.co/datasets/OpenPipe/hacker-news and, honestly, Iām not really sure how this was obtained, if using scarping or if HN makes this data public in some way."