It always confuses me as well (and I have at least some Physics background, having done a year as an undergrad). I think this is because I confuse it with the statistical measure of the entropy of a distribution, where a uniform distribution will maximise entropy.