The most obvious problem with such a system is that you're a bad person regardless of what you do, so there is no value in doing good unless you're a superintelligence. You say as much above- people make immoral decisions (by your evaluation) constantly, so it doesn't really matter whether you do.
You seem to be angling above for a world where people who care about others feel constant shame because they're not doing enough, while selfish, shameless people get to lead happy lives. That seems immoral, even by my much lower standards.
I agree that labelling people as good or bad is a mistake. In fact, my issue here is that you explicitly labelled everyone as a bad person. Is this not what you meant to say?
I don't believe I said your conclusion makes me unhappy, my intent was to say that your conclusion is immoral and, as a moral position, it is therefore wrong.
Trying not to do harm would be an easier goal to define.