News for hackers. Stuff that matters.

A comment on Kullback–Leibler Divergence. Parent comment

Re:Kullback–Leibler Divergence by ramblenode on Tuesday August 22, 2023 @08:30PM (#37230414)
> If you have two independent events it makes sense that the total surprisal (information content) should be the sum of the surprisals of the two events.

I think the parent is also asking why we would expect surprisal to be additive rather than multiplicative like probabilities.

[ Parent | on HN ]
Re:Kullback–Leibler Divergence by Majromax on Tuesday August 22, 2023 @09:04PM (#37230649)
Because if two things happen – one totally expected and one very surprising – then on net you’re still surprised.
[ Parent | on HN ]
Re:Kullback–Leibler Divergence by ramblenode on Wednesday August 23, 2023 @12:37PM (#37238641)
That's also the case for probabilities.
[ Parent | on HN ]
Re:Kullback–Leibler Divergence by joelthelion on Wednesday August 23, 2023 @04:36AM (#37233311)
I think in part because it's really nice to measure information in bits rather than tiny probabilities. And it aligns well with how information is stored.
[ Parent | on HN ]