The idea extraction step is simple when starting from scratch, but when you got megabytes of text deduplication becomes a problem. I like to circle back a lot.
What I'm experimenting with is building a graph out of "entities", which I'm calling people, places and things found in each card. The point I guess is to allow the LLMs to make connections between cards, but this won't be replacing user made connections.
This lets me pick up all cards that reference something like, I don't know, "factories" as a concept, which isn't something the user will necessarily want to directly connect.
Downside is that deduplicated taxonomy: I have, from a scan through my list, "factories", "Factory" and "Factories". My attempt to head this off is by vector searching based on the proposed entity keywords and having an LLM decide if they are the same or not. I think I need to work on my prompt for this though.
It's good for bridging the gaps between LLMs, ontologies, traditional structured data, and logic programming.
I run into the same problem. I made a RAG system and imported 15 years of reddit and HN content (my own message logs) + 2 years of LLM chats, totaling about 80MB of text. I can use it to retrieve fragments but there is duplication, and almost all duplicate fragments have slightly different approaches. How do I merge all of them, and how do I get a deduplicated taxonomy? I got about 290K unique keywords extracted from the text, it doesn't fit into LLMs.
I am gravitating towards building graphs of ideas, and having a way to generate unique (non duplicate) new ideas while ingesting new text.