It is the simplest and only way to make them competitive for an investor. Anyone can buy them (that can invest in the U.S. anyways). If you tax them, nobody would buy them (risk versus earnings). To become a competitive and practical choice, and since it is an option for this investment, they make them tax free. Otherwise they would have to increase the interest to an unrealistic amount; at that point there would be no lending option practically. (Yes, investors only buy municipal bonds for the tax free trait to be honest. Not a loophole, just giving your money earlier to the government if you think about it from an opportunity cost perspective.)

Think of it as a way to take money from state and federal taxes and to move them to municipalities etc. It's either that or the government above would have to pick up the slack.