Re:Show HN: Search HN for interesting comment sections
by humbleferret on Tuesday April 16, 2024 @09:32AM
(#40051636)
Good work!
The sample searches were particularly strong for uncovering discussions on specific subjects, i.e. 'Dedicated Vector Databases'.
With less focused searches, such as 'Deep Work', I notice some results feel only loosely related. Is this an aim – to broaden the scope of discovery that may not seem directly related?
Re:Show HN: Search HN for interesting comment sections
by isoprophlex on Tuesday April 16, 2024 @09:35AM
(#40051671)
Thanks for testing it out! So, because I've found (in purely vibe-based testing, of course) that with queries of 1, 2 or 3 words, a vector embedding doesn't always make a lot of sense. What's "Apple"? The fruit, the company? So, here it defaults to priming the fuzzy search with a full text search on 'apple', or 'flow state'. If you want to know something about deep work, try "tricks for performing deep work", or "help me get into a flow state".
It's all very serendipity-driven, and this is mostly a tool for wasting time, but you might discover some interesting (and hopefully on point) conversations this way.
Anyway, valuable feedback. Maybe in a next interation, I'll always do both searches and merge the resulting ranking.
Re:Show HN: Search HN for interesting comment sections
by gus_massa on Tuesday April 16, 2024 @09:38AM
(#40051699)
Feature request: I expected that clicking on the number of comments would send me to the HN page like in the HN search. Just make all the second line clicky too.
Re:Show HN: Search HN for interesting comment sections
by isoprophlex on Tuesday April 16, 2024 @09:19AM
(#40051525)
I built this tool to help me find interesting discussions on Hacker News. I love reading HN discussions almost more than the articles themselves. However, I found that full text search, although highly performant, is not always good at surfacing interesting discussions on a certain topic -- especially if you don't know what to search for exactly.
I built this by scraping the most recent ~6 million posts (that's about 2 years of history) and putting the resulting posts and their vector embeddings into Postgres.
Let me know what could be improved, and if you'd like a more detailed writeup of how this was built :)
Re:Show HN: Search HN for interesting comment sections
by isoprophlex on Tuesday April 16, 2024 @02:43PM
(#40055682)
Yes, exactly! Your feedback prompted me to do a quick iteration into something good enough for a Show HN, so thanks :)
I see what you mean about the margin on mobile... note that I'm one of the most unqualified front end people I know and this was mostly done using github copilot, I might not be brainy enough to understand the intricacies of css responsive layouting ;)
Re:Show HN: Search HN for interesting comment sections
by ryandrake on Tuesday April 16, 2024 @03:24PM
(#40056112)
Minor nit-pick: The browser back button doesn't seem able to navigate back to the landing page.
Interesting project, though. As a shortcut, I usually just use the hidden /active URL to find the interesting stories. Generally these are the ones with the most active commentary and often they get pushed off the main HN screen quickly by the "flamewar detector". So you have to be quick to find them, or just use /active.
Re:Show HN: Search HN for interesting comment sections
by isoprophlex on Tuesday April 16, 2024 @03:43PM
(#40056321)
Yeah, good observation, that really annoyed me too. I tried to get HTMX to mess with the browser history, but that only seemed to make things worse...
Interesting aside on flamewar detection: you can visualize the comment embeddings in a 2d scatterplot, color coded by username. As you'd expect, a great deal of dang's comments occupy a distinct position in comment-space.
Re:Show HN: Search HN for interesting comment sections
by paulgerhardt on Tuesday April 16, 2024 @08:46PM
(#40059183)
I’d love to see the default examples seeded from the “best comments” section [1].
Three of examples on the current home page surface some toxic threads (“llm waifus”, “internet of shit”, and “AI Doomers”)… which while controversial aren’t as rich or insightful as say the comments from people that built Rust, the whole earth catalog, or the first x86 chip.
The sample searches were particularly strong for uncovering discussions on specific subjects, i.e. 'Dedicated Vector Databases'.
With less focused searches, such as 'Deep Work', I notice some results feel only loosely related. Is this an aim – to broaden the scope of discovery that may not seem directly related?
It's all very serendipity-driven, and this is mostly a tool for wasting time, but you might discover some interesting (and hopefully on point) conversations this way.
Anyway, valuable feedback. Maybe in a next interation, I'll always do both searches and merge the resulting ranking.
I built this by scraping the most recent ~6 million posts (that's about 2 years of history) and putting the resulting posts and their vector embeddings into Postgres.
Let me know what could be improved, and if you'd like a more detailed writeup of how this was built :)
Also, somehow this has a relatively big horizontal margin on mobile.
I see what you mean about the margin on mobile... note that I'm one of the most unqualified front end people I know and this was mostly done using github copilot, I might not be brainy enough to understand the intricacies of css responsive layouting ;)
I'll post a writeup sometime too!
Interesting project, though. As a shortcut, I usually just use the hidden /active URL to find the interesting stories. Generally these are the ones with the most active commentary and often they get pushed off the main HN screen quickly by the "flamewar detector". So you have to be quick to find them, or just use /active.
Interesting aside on flamewar detection: you can visualize the comment embeddings in a 2d scatterplot, color coded by username. As you'd expect, a great deal of dang's comments occupy a distinct position in comment-space.
Three of examples on the current home page surface some toxic threads (“llm waifus”, “internet of shit”, and “AI Doomers”)… which while controversial aren’t as rich or insightful as say the comments from people that built Rust, the whole earth catalog, or the first x86 chip.
[1] https://news.ycombinator.com/bestcomments