Take it further: the IRS just gives you a form at the end of the year saying your income was $X and your taxes withheld were $Y. Sign here to get a refund or pay the shortfall (which, if withheld correctly, there should never be a shortfall).
That's actually how it works in most developed countries.
As for small businesses (the vast majroity of non-W2 employees), sure, audit them if they have been running a restaurant for years and weirdly have never drawn an income.
Furthermore, the system doesn't work too well with variable incomes (it's based per-check) and should two married people both have variable incomes it can't work, period. (We are both self-employed--and some years ago I threw in the towel on even trying to get it right. I simply stay at the safe harbor limit. Come Apr 15 I expect to be off by 4-digit amounts, but it could be in either direction.)
And your threshold won't catch most cheating. Not drawing an income will of course be suspect, but the guy who reports $50k but took $30k in cash out of the till over the year won't show up on your radar. I actually get a 1099 but my wife does not--no computer can know if she's reporting accurately. They would have to do a bit to even realize she's working (she's past retirement age.)
... how long do you think it would take for everyone of those W2 employees to be filing for the Foreign Tax Credit every year if it were illegal to audit them?
Why would anyone who made under $50,000 or $100,000 ever bother to pay one cent of taxes if you did that?