Maybe the problem is that they are a s*ty agency that randomly fines people and expect them to pay up. I don't have knowledge how the IRS works but I know at least one tax agency in this world that would fine you random amounts and just wait for your reaction. They are not in the business of fairly auditing your numbers because they'll lose, so they turn to poorer persons that they can bully.

Another thing, many countries expect rich people and rich companies to hire government-certified accountants to verify your fillings (known as a Statutory auditor https://en.wikipedia.org/wiki/Statutory_auditor ). This makes it harder for these companies to do stuff the non-audited guy can easily do.