Body armor is heavy and obvious, especially in warm climates. I don't think it would start an arms race. You would only wear it when you are attacking or expecting an attack. Not to mention it only protects your chest/back.

Body armor comes in a wide variety of sizes and weights many of which are not heavy and obvious. You don't need a full getup with plates to stop handguns.

Very true.

Not only that but it makes it easier for criminals to have protection from police bullets.

I don't think criminals are wise, but protection from police bullets doesn't seem like a reasonable motivation to wear bulletproof clothing. If required, the Police will simply call out a SWAT team using military weapons. There should be no reasonable expectation that somehow you can survive an armed exchange with the police.

Sadly, there are criminals that have tried bulletproof clothing with the resulting escalation of violence. In 1986 bank robbers wearing bulletproof outfits wounded five and killed two FBI agents in the 1986 FBI Miami shootout. Naturally, the two bad guys didn't survive. See [1].

[1] https://en.wikipedia.org/wiki/1986_FBI_Miami_shootout