I think we don't hold them accountable for the same reason we generally accept panhandlers (which is usually technically illegal in most jurisdictions). If we hold them accountable, where do we send them? Jail? Does that actually fix the problem, or does it just hide it under a societal rug?