I think in part because it's really nice to measure information in bits rather than tiny probabilities. And it aligns well with how information is stored.