It's not necessarily unhealthy to an individual person, it's the effect that it has on the surviving bacteria. These soaps are dangerous to society as a whole because they become less effective the more widespread their usage is [0]. Thus, we don't need to use them for every single thing we get on our hands that we want off, we're not in mortal peril of those germs.

[0]: http://www.fda.gov/ForConsumers/ConsumerUpdates/ucm378393.ht...