Because this year I got hit by some underpayment penalties (with interest added on top) but at the same time I also got a refund which had no interest applied to it.
That is being self employed and paying estimated taxes every quarter.
I realized that several years back, I had worked at a temp job for a very brief time and I probably never got a W-2 and therefore never paid taxes on it. Thus, I called up the IRS, got the W-2 information which they had, and duly filed an amended return with a check for something like $20.
However, in a rather short amount of time, I got a letter from the IRS stating that they could not accept a payment for a return more than a couple years old and here was a check for the amount, plus interest.
So I can tell you they not only pay interest on your money, but they pay it on money they are returning even though you actually owed it.
Also, it sounds like you're referring to the penalty for underpayment, which I believe is currently 3.6% of the total underpayment for the year [0]. This only kicks in under certain circumstances.
[0]: https://www.irs.gov/pub/irs-pdf/f2210.pdf