Another way to see it is as a continuous generalization of the idea that you need n bits to represent 2^n equally likely alternatives.
I'm stuck on why we care about surprisal as `log 1/p`.
If your distribution has N equally-likely values, `p(x) = 1/N`, and information/surprisal `I(x) = log(N)`. In base 2, this is how many bits are required to specify exactly WHICH of the N values you're talking about.
If `x` is not a single one of the N states but an event consisting of `n(x)` states, then `I(x) = log(N) - log(n(x))`, suggesting it takes _somewhat less information_ to specify this particular state, and it correctly gives 0 if `n(x) = N`, i.e. there's only one state.
Exactly what this "less information" means is vague, but you might think of it in terms of compression: if you compress some stream of data which is sampled from `X` with probability `p(x)`, you could use use the shortest codes (0, 10, 11, etc) for the most common values with some "stop word" to say when the end of a datum is reached. `I(x)` captures this sense in general, but it might only become literally true in the limit of a very large stream of data with a very large dictionary.
I think the parent is also asking why we would expect surprisal to be additive rather than multiplicative like probabilities.
Times a constant!
Hartley was (as far as I know) the first person to recognise the usefulness of log probabilities in the context of measuring information [2]. It's a really amazing paper, for several reasons ... but one thing that really strikes me about it is how it's aged: the first half has an essentially timeless presentation of the essence of information, and the second has a now-quite-irrelevant presentation of how to make a better TV. I guess that was the really interesting problem at the time!
[1] https://people.math.harvard.edu/~ctm/home/text/others/shanno...
[2] http://dotrose.com/etext/90_Miscellaneous/transmission_of_in...
Good question. It's because -log P(X) is the number of bits of information I would need to eliminate any surprise about the value of X.
For example, let's say X is the value of an 8-bit register, and each of the 2^8=256 possible values has a probability of 1/256. I would need to learn all 8 bits to know the value of X: so Surprisal(X=x) = -log P(X=x) = -log 1/256 = 8.
Suppose I learned only that X is odd, which is equivalent to learning that the last bit of the register is equal to 1. The probability that X is 1 is 1/2, and so I would need to be provided with -log P(X is odd) = -log 1/2 = 1 bit of information to learn that X is odd.
Information can be thought of as simply the elimination of uncertainty. So Surprisal (uncertainty) is basically a measure of information, but instead of how much information you have, it's how much information you need to eliminate all uncertainty.
Log shows up a lot but I don't think I ever really understood when/how do people decided to use it in their formulas. One KL divergence youtube video said, `let's normalize the p(P)/p(Q) using (log(p(P)/q(P)))^N` and then shows how to derived the KL formula. I'd also appreciate if you know how the N (sample size ig) is being used here.
For a lot of math stuff I endup being confused about how people use operations. Like using 1/p(X=x) is like magic to me because I don't understand what the context is when someone thinks about the problem and then decides to do something. What's their thinking process here, what tool or process do I not know about that makes me confused?