News for hackers. Stuff that matters.
Re:Knowledge Compressor by Mappleton on Tuesday August 25, 2026 @12:41PM (#49436883)
How far can documentation be compressed without losing its meaning?
Re:Knowledge Compressor by neoncontrails on Tuesday August 25, 2026 @01:22PM (#49437468)
This is a good question! The answer is, unfortunately, uncomputable – or so we think. But if you manage to find a more satisfying answer, you'll have solved one of the more beguiling problems of the last century, which is the basis for most of modern Information Theory. You can learn more about it here: https://homepages.cwi.nl/~paulv/kolmogorov.html
Re:Knowledge Compressor by idan on Tuesday August 25, 2026 @01:23PM (#49437489)
Hello! Idan (from Next) here, and others from the team will drop by the thread if you have questions!
Re:Knowledge Compressor by adsharma on Tuesday August 25, 2026 @05:45PM (#49441122)
Did you consider extracting knowledge from the doc into a graph database (while passing the Q&A) and compressing the graph instead?
Re:Knowledge Compressor by adsharma on Tuesday August 25, 2026 @07:18PM (#49442023)
https://huggingface.co/datasets/ladybugdb/github-knowledge-c...

These methods could not beat the 480 token result because of the small dataset. But for a sufficiently large corpus with graph reordering and columnar parquet compression, they could be quite competitive.

The parquet files are self explanatory. For knowledge.lbdb.zst, first decompress with zstd and then run a cypher such as:

  lbug> MATCH (a:E)-[t:TRIPLE]->(b:E) RETURN a.name AS subj, t.predicate AS pred, b.name AS obj, t.props AS props, t.scope AS scope;