A trail that hits that balance well IMO is https://trails.pieterma.es/trail/pacemaker-principle/. I find the system theory topics the most interesting. In this one, I like how it pulled in a section from Kitchen Confidential in between oil trade bottlenecks and software team constraints to illustrate the general principle.
There's two stages to the linking: first juxtaposing the excerpts, then finding and linking key phrases within them. I find the excerpts themselves often have interesting connections between them, but the key phrases can be a bit out there. The "fictions" to "internal motives" one does gel for me, given the theme of deceiving ourselves about our own motivations.
I think LLMs are overused to summarise and underused to help us read deeper.I built a system for Claude Code to browse 100 non-fiction books and find interesting connections between them.I started out with a pipeline in stages, chaining together LLM calls to build up a context of the library. I was mainly getting back the insight that I was baking into the prompts, and the results weren't...
LLMs are often used to reduce the amount of reading we do. I’m hopeful they can be used to augment our reading habits instead.I gave Claude Code a set of CLI tools to explore a library of 100 non-fiction books (sampled from HN’s favourites) and surface sequences of excerpts that link ideas.I was first intrigued when it linked Steve Jobs’ self-deception to an excerpt about Theranos, and it ended...
I scraped HN's 1000 most mentioned books and visualised them. This month I used a new embedding model (Nomic), switch out UMAP for PaCMAP, and added automatic cluster labelling.
The clustering and dimensionality reduction aren't quite as stable as I'd like, but most seeds give decent results now.
You've got the cluster semantics spot on, to be honest. Broad genres are grouped together, with a tendency for sub-genres to be grouped locally within those.
There is no interpretation of the overall shapes or the global structure, those are more a result of a particular UMAP run than inherent in the data.
Would love to provide different views on it and go more in depth next, thanks for the suggestion.
The cluster memberships that come out of the first round are distributions over the different clusters, e.g. a given book is weighted 0.8 for cluster A and 0.2 for cluster B. The Hellinger distance is well-suited to quantify the difference between two distributions like that. Cosine similarity and Euclidean distance worked as well, but Hellinger gave subjectively nicer results.
Very interesting question, I'm not sure! While developing, I noticed that the systems thinking books were spread over different genres, which I found quite pleasing. However, I'm not sure if other books were even more diffuse. I'll have to dig back in and find out :)
Hey HN! I love finding new books to read on here. I wanted to gather the most mentioned books and recreate the serendipity of physical browsing. I scraped 20k comments from HN threads related to reading, extracted the references and opinions using GPT-4o mini, and visualised their embeddings as a map.- OpenAI's embeddings were processed using UMAP and HDBSCAN. A direct 2D projection from the...
Hey HN! I love finding new books to read on here. I wanted to recreate the serendipity of browsing someone else's bookshelf. I scraped 20k posts from HN threads related to reading, extracted the references using GPT, and visualised their embeddings as a map.- OpenAI's embeddings were processed using UMAP and HDBSCAN. A direct 2D projection from the text embeddings didn't yield visually...