News for hackers. Stuff that matters.
Re:Information Theory with TensorFlow by yters on Sunday December 15, 2019 @11:53AM (#21797166)
The significant thing is not that we're arbitrarily applying the log function to probabilities. It is that there's a relationship between expected code length and the entropy of a source. It is surprising, to me at least, that the length of a thing is related to its probability. From that relationship come a number of fascinating connections to many other fields. Check out Cover and Thomas' "Elements of Information Theory" for a very approachable introduction to all these connections.
Re:Information Theory with TensorFlow by jbay808 on Sunday December 15, 2019 @02:27PM (#21798348)
Isn't that exactly what Occam's Razor teaches us?

"Entities should not be multiplied without necessity."

Re:Information Theory with TensorFlow by yters on Sunday December 15, 2019 @03:57PM (#21798929)
Yes info theory is a formalization of Occam's razor