Re:Knowledge Compressor
by adsharma on Tuesday August 25, 2026 @05:45PM
(#49441122)
Did you consider extracting knowledge from the doc into a graph database (while passing the Q&A) and compressing the graph instead?
[ Parent
| on HN ]
https://huggingface.co/datasets/ladybugdb/github-knowledge-c...
These methods could not beat the 480 token result because of the small dataset. But for a sufficiently large corpus with graph reordering and columnar parquet compression, they could be quite competitive.
The parquet files are self explanatory. For knowledge.lbdb.zst, first decompress with zstd and then run a cypher such as:
lbug> MATCH (a:E)-[t:TRIPLE]->(b:E) RETURN a.name AS subj, t.predicate AS pred, b.name AS obj, t.props AS props, t.scope AS scope;