Re:Dictionary Compression
by lifthrasiir on Tuesday March 01, 2022 @09:03PM
(#30522764)
You calculated the sum of entropys of each word, not the entropy of the entire word list as a whole (which would be much lower due to the cross-correlation).
https://gist.github.com/cfreshman/a03ef2cba789d8cf00c08f767e...
Here is the code:
This is a textbook entropy implementation and I've been under the impression that you'll never do better than this (for a given word size).