Re:Shannon’s Source Coding Theorem (Foundations of Informati...
by rododendro on Sunday June 18, 2023 @11:55AM
(#36381380)
Well in this case the distribution would simply be a uniform distribution over [1:6], which has entropy 2.58 for each symbol. Which is also the most entropic distribution on the sets of size 6.
Like other have said, you can't actually know in a real world scenario your underlying distribution in whatever file you are trying to compress, but since you want to compress it you are telling the program that the file has interesting information (otherwise it would be just noise) and so grammar based encoding would probably not be best, instead lz77 or lz78 would perform better (for example, if you know more about the file you could even do better by transforming first into some frequency domain and then compressing).
Re:Shannon’s Source Coding Theorem (Foundations of Informati...
by nico on Sunday June 18, 2023 @12:48PM
(#36381902)
Fascinating. Thank you.
Why do you need to know more about the file to transform to frequency domain?
Re:Shannon’s Source Coding Theorem (Foundations of Informati...
by rododendro on Sunday June 18, 2023 @01:19PM
(#36382236)
Because no information is lost if you simply change domain, but you may spot more patterns faster or easier. For example (even if a bit dated) https://ieeexplore.ieee.org/abstract/document/1614066
There is a lot of research in this field because of the IoT fad/trend. If you want to know more i would suggest to simply pic a paper from IEEE and deep dive into the terms and terminology and simply skip the proofs of most algorithms/theorems.
In any case, you don’t need to index the whole numbers space to transmit all the information
Like you say, you can use 13 bits instead of 15
And using variable length encoding you can reduce it even more
But what is the distribution in this case?