News for hackers. Stuff that matters.
Re:I want everything local – Building my offline AI workspace by oblio on Friday August 08, 2025 @05:25PM (#44841849)
It feels weird that the search index is bigger than the underlying data, weren't search indexes supposed to be efficient formats giving fast access to the underlying data?
Re:I want everything local – Building my offline AI workspace by andylizf on Friday August 08, 2025 @05:29PM (#44841882)
Exactly. That's because instead of just mapping keywords, vector search stores the rich meaning of the text as massive data structures, and LEANN is our solution to that paradoxical inefficiency.
Re:I want everything local – Building my offline AI workspace by yichuan on Friday August 08, 2025 @05:37PM (#44841948)
I guess for semantic search(rather than keyword search), the index is larger than the text because we need to embed them into a huge semantic space, which make sense to me
Re:I want everything local – Building my offline AI workspace by iezepov on Saturday August 09, 2025 @12:26AM (#44844030)
Good point! Maybe indexing is a bad term here, and it's more like feature extraction (and since embeddings are high dimensional we extract a lot of features). From that point of view it makes sense that "the index" takes more space than the original data.
Re:I want everything local – Building my offline AI workspace by catlifeonmars on Saturday August 09, 2025 @01:06AM (#44844160)
Why would the embeddings be higher dimensionally than the data? I imagine the embeddings would contain relatively higher entropy (and thus lower redundancy) than many types of source data.
Re:I want everything local – Building my offline AI workspace by cm228 on Saturday August 09, 2025 @06:07AM (#44845319)
depends on the chunk-size used to create the embedding.
Re:I want everything local – Building my offline AI workspace by brookst on Saturday August 09, 2025 @09:49AM (#44846430)
Nonclustered indexes in RDBMS can be larger than the tables. It’s usually poor design or indexing a very simple schema in a non-trivial way, but the ultimate goal of the index is speed, not size. As long as you can select and use only a subset of the index based on its ordering it’s still a win.